If you're looking into picking up some Indian groceries or sweets then Journal Square might be an interesting place to check out.
However most of the interesting stuff in Jersey city is around Van Vorst Park or Hamilton Park. These areas are pretty easily accesible from the Grove St PATH station. Jersey City is supposed to be the "new Williamsburg" where all of the artists and hipsters are hanging out. This is the area you should check out if you are interested in this.
Also be sure to check out the area along the waterfront. Exchange Place in Jersey City is alright, but I think Hoboken is much more interesting.
In Hoboken they have a little promenade along the waterfront which is nice, and has an excellent view of the city (especially at night). Hoboken also has a good number of interesting bars, which are worth checking out.