Python websocket multithread/ multiprocess storing the received JSON data in one thread questions

tobywong
tobywong
in WebSocket API

For my development, I’m trying to have two threads.

One threads is for receiving real time data and put them into a list/dictionary(which one should I use to store the JSON I received?)

And the other thread is to loop through the list/dictionary and print out the data in the JSON object/string we need.(not all the fields are needed in the received JSON data)

But the second thread doesn’t start.

The reason to use multithread is because we don't want to miss out on updated data when the program is printing the data(eventually, the "print" action will be replaced by updated our database)

What is the problem?

I could provide my code if it's needed.

Thank you.

Tagged:

Best Answer

  • nick.zincone
    nick.zincone
    Answer ✓

    Hi @tobywong

    You can probably search on the internet for multithreaded examples in Python that suit your requirements. When I do this, I explicitly search for an example to achieve multi-threaded processing.

    For example:

    # Example threading using the thread-safe 'queue.Queue' class which allows you to put
    # data into the queue on one thread and retrieve it on another - asynchronously

    import threading
    import queue

    # Create the queue
    data_queue = queue.Queue()


    In thread 1, when you receive your data, simply store it within the queue

    def receive_data_cb(data):
       data_queue.put(data)

    In thread 2, process the data when it becomes available

    def process_data():
       while True:
          data = data_queue.get()    # Block until data arrives
          print(data)                # This is where you can store in a DB

    To put it all together, create 2 threads and start each:

    thread1 = threading.Thread(target=receive_data_cb)
    thread2 = threading.Thread(target=process_data)

    thread1.start()
    thread2.start()

    The above is the general algorithm I found by researching. However, I would personally experiment and try some simple tests. For example, you may not need to explicitly create 2 distinct threads here but instead just one which handle your processing of the data. The main thread, where you start your application, could handle the data coming from the WebSocket server.

Answers

  • tobywong
    tobywong

    Using queue.Queue works! Thank you so much. But why doesn't list work?

  • pf
    pf

    Hi @tobywong

    Queue class is much more complex than list and designed to operate efficiently in multi-thread context.
    This article compares list and Queue: https://www.troyfawkes.com/learn-python-multithreading-queues-basics/

    Behind the scene, Queue is managing a deque or a list, but if you check Queue.put() and Queue.get(), you'll see that these functions are using:

    • a Semaphoe to protect simultaneous call to both functions
    • wait() function to let the GIL to suspend the current thread and let others restart

    With your implementation, if you don't use async function, the thread that manages the list can't be suspended by the GIL so others threads are waiting infinitely to run

Categories