examples.txt

#my plans {have a search engine that can fine stuff in a query }

 #example documten{
 #{"id": "document"
 #"text": """this is a blop of text that has blah blah blah words and some words, its a typical document"
 #"created: "2012-02_18T20:18:00-0000
 
 #tokenization [taking our big document text blob and breaking into usable word sizes{filtering meaningless words using text blob spliting white space.}] 
 
 #stemming it tokenizes stemming finds the root words  (avoid manually searching through the whole blob... )
 
 #ngrams solves most of the shortcomming of stemming 

 #inver index is key value its terms we just got from edge n grams  
 
 ##sudo index = {
     # 'blob' : {
         # 'document-6969': [3],
         # },
         # 'text':{
             # 'document-6969' : [5, 10],
             # },
    #}

 #segments (lucen java )
     always sorted 
     json for postion and documents 


     querys break into 3 steps 
        query parser 
        index reader
        scoring

            query 
                users info -> 
                stop_words= set([
                    'a' , 'an', 'and', 'as', 'are', 'at', 'be', ''''''''''''''''''''''''''
                ])

                def parse_query(query):
                    tokens = [token for token in query.split()if not token in stop_words ]
                    return make_ngrams(tokens)