AsadiAhmad/Ngram-Spark-Wikipedia
Calculating Ngram with PySpark for wikipedia text
This project helps you analyze large collections of text, specifically Wikipedia articles, to understand common word patterns. It takes in raw text data from Wikipedia and outputs lists of frequently occurring word sequences (like 'New York' or 'machine learning'). Anyone working with large text datasets, such as researchers, linguists, or data analysts, can use this to extract insights into language usage.
No commits in the last 6 months.
Use this if you need to quickly identify and count common phrases and word combinations across massive text corpora like Wikipedia.
Not ideal if you're looking for advanced sentiment analysis, entity recognition, or more complex natural language processing tasks beyond basic n-gram calculation.
Stars
29
Forks
—
Language
Jupyter Notebook
License
MIT
Category
Last pushed
Jun 03, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/AsadiAhmad/Ngram-Spark-Wikipedia"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
dipanjanS/text-analytics-with-python
Learn how to process, classify, cluster, summarize, understand syntax, semantics and sentiment...
jonathandunn/text_analytics
Basic text analytics and natural language processing in Python
IBM/watson-document-co-relation
Correlate text content across documents using Watson NLU, Python NLTK and Watson Studio.
Clarifai/clarifai-pyspark
Interfaces for Unstructured data and ML pipelines with Databricks and Clarifai
umer7/Applied-Text-Mining-in-Python
Repo for Applied Text Mining in Python (coursera) by University of Michigan