- Here Is A Small Example Content Of The Docword Txt File Docld 3 3 2 1 2 2 3 3 3 1 1 1 Vocabld 3 3 2 5 2 1 1 15 14 3 1 C 1 (23.08 KiB) Viewed 72 times
- Here Is A Small Example Content Of The Docword Txt File Docld 3 3 2 1 2 2 3 3 3 1 1 1 Vocabld 3 3 2 5 2 1 1 15 14 3 1 C 2 (63.74 KiB) Viewed 72 times
Here is an example of the vocab.txt file vocabld 1 2 3 4 5 | دي | | C word plane car motorbike truck boat
b) (spark SQL] Create a dataframe containing rows with four fields: (word, docid, count, firstLetter). You should add the firstLetter column by using a UDF which extracts the first letter of word as a String. Save the results in parquet format partitioned by firstLetter to docwordIndexFilename. Use show() to print the first 10 rows of the dataframe that you saved. So, for the above example input, you should see the following output (the exact ordering is not important): 1 word docidi count first Letter --- plane! 11 10001 1 plane! 31 1001 1 car! 215001 carl 11 1201 motorbike! 11 12001 I motorbike 217021 motorbike 316001 1 truck 31 1221 boat! 31 20001 boat! 21 2001 bl PI