Most blog posts are about simpler controls or regulations that a microcontroller handles. But the IoT devices, short for Internet of Things, can do much more. The basis for this is provided by protocols that ensure communication to the individual devices, such as the MQTT used here. This first post of the blog series about MQTT teaches the basics. At the end of the blog series, a small robot is controlled using MQTT, similar to a remote control.
You only need a few components for this blog post, see Table 1.
|1||Matching power supply|
Table 1: Required hardware
With the Pi, keep in mind that in addition to the hardware mentioned above, you'll also need a microSD card. For this you should install Raspberry Pi OS (formerly Raspbian) as an image on the card.
Hint at this point
This tutorial covers MQTT and how to set up your own MQTT broker and clients. Therefore, only for this part you need a Raspberry Pi to follow and copy all the steps shown here. In the further parts of the blog series more clients will be integrated, like the microcontroller board compatible with Arduino Uno R3 and NodeMCU's.
What is MQTT?
MQTT is the abbreviation for Message Queuing Telemetry Transport and is also known under the older name WebSphere MQTT, WSMQTT for short. It is an open network protocol for machine-to-machine communication, e.g. to transmit sensor data and/or to control actuators. The background of this protocol is that a transmission of information can be guaranteed even in so-called unstable networks. The term "unstable networks" originates from the time when networks were still very susceptible to interference.
MQTT, if installed by default, is accessible via network port 1883. The second port 8883 is the secured network port, but must be encrypted using the TLS protocol. In order to use MQTT, you need an MQTT server, the so-called broker, as well as the corresponding end devices that send or receive the data, also called clients. Usually the broker is found on a Linux-based operating system. However, Windows operating systems are also possible. Meanwhile, MQTT can be found in automation technology and IoT projects, since it is relatively easy to set up an MQTT network with few resources.
The protocol version 3.1.1 used here, as well as the previous version 3.1, were published in 2010 under a free license, which was specified by the OASIS standardization committee. This protocol version is therefore already over 10 years old at the time the blog was published. The latest protocol version 5.0 was published in 2019 and at the same time specified by the OASIS standardization committee.
How does MQTT work
MQTT uses the event-driven publish/subscribe architecture. There is no end-to-end connection, as for example with a website, but a server, the so-called broker, to which senders and receivers, the so-called clients, connect. In order to better understand these connections, Figure 1 illustrates this in a simplified form.
Figure 1: Illustration of the MQTT architecture
A client, in this case the sensor, the mobile device or the IoT device, can always send and/or receive data; the broker merely serves as an information buffer that forwards the data to other clients. In order for the client to receive a corresponding message from the broker, the client must subscribe to the so-called topics when the connection is established or during operation at the broker.
You can think of a Topic as a kind of URL (Uniform Resource Locator) that reflects the desired content. A topic for a temperature sensor in the living room could publish the measured values on Home/Ground floor/Living room/Temperature. To be clear right away, it is not an address to the sensor, but allows access to the last transmitted sensor data. When the reading is updated, the broker checks which client has subscribed to receive this information and forwards it.
Data exchange with the broker
As mentioned at the beginning, MQTT was intended for unstable connections, which is why there are three different service qualities, Quality of Service or QoS. With level 1 or QoS = 0, the message is sent exactly once. An acknowledgement, as is the case with the other QoS, is completely ignored. Only the publishing, the so-called PUBLISH, is important.
With QoS = 1, one also speaks of "delivered at least once", which means that the sender waits for an acknowledgement from the receiver. This acknowledgement is called PUBACK and is obligatory for the remote station, otherwise the sender transmits until an acknowledgement is received. This leads in reverse to the fact that a message is transmitted several times to the receiver.
QoS = 2 is the combination of QoS = 0 and QoS = 1, whereby it also represents the slowest transmission. Here the message is sent exactly only once. To achieve this, a two-stage confirmation of receipt is necessary.
First, the sender sends (PUBLISH) its message to the recipient, which accepts the message and sends a confirmation message (PUBREC), with the content from the sender attached as a copy. If this confirmation message (PUBREC) is received by the sender, the latter saves the information and also replies with a confirmation message (PUBREL).
Finally, when the recipient has received the PUBREL, this sends the last confirmation message (PUBCOMP). At the end of such a transfer of data, it is ensured that the message has actually arrived. To keep it more understandable, shows Figure 2 once again schematically how the message flow looks with all three QoS variants.
Figure 2: MQTT QoS overview
Subscribe to data from the broker
Now the question arises as to how a client can now query sensor data? There are again several variants that can lead to the goal. In the first variant, as already shown above for the simple sensor data transfer, you can specify the topic completely. To do this, simply enter the topic Home/Ground floor/Living room/Temperature. However, you will then only receive exactly this one value.
When using variant two, the + operator comes into play. With Home/+/+/Temperature you get the temperature data for all rooms on each floor. Useful I this if you want to predefine such a structure for all sensors in their house in such a way and display e.g. bundled.
Variant three goes one step further with the # operator. By Home / Ground Floor / # get all the information available for each room on the ground floor. Depending on the quantity of information, a lot can come together at this point.
The fourth and last case is probably the most data-intensive topic subscription, if you send a lot of data. With /# you subscribe to the root directory and every change is sent to you. I do not recommend this variant at this point! You should think in advance about your structure or the measurement data you want to include and define it beforehand.
Finally, the question probably arises, what happens if a transmitter suddenly stops transmitting? For this purpose, there is the so-called retained message in MQTT, which contains the last will and testament. In order for this to be used, the sender, in our case a sensor, must provide this retained message when connecting to the broker. This stores what is to be sent if the sender can no longer be reached. An individual content can then be written by the broker for each topic. Since MQTT already provides monitoring by default, you do not need to worry about this point. Since it goes too deeply into the matter, I can give at this point however gladly the keyword LWT message, with which you will find enough reading material in the Internet.
The Raspberry Pi becomes a broker
As mentioned in the “What is MQTT?” Section, this protocol is very popular in the IoT scene. Many shy away from using their personal data or data to control their home on the Internet, for example mqtt-dashboard.com/ save. Therefore, it makes sense to set up a small, local broker with the Raspberry Pi. At first it does not matter which Raspberry Pi version you are using and whether it is already taking on any tasks.
First of all, open the terminal of your Raspberry Pi, see Figure 3 and then issue the commands Code 1 a.
Figure 3: Open Terminal in Pi
sudo apt upgrade
sudo apt dist-upgrade
Code 1: update Raspbian
With this you update the Raspberry Pi and all installed packages. Then issue the command Code 2 a. This will load the popular mosquitto broker and the client application onto your Raspberry Pi. Confirm the installation in the terminal at the appropriate point.
sudo apt install mosquitto mosquitto clients
Code 2: Install Mosquitto server and client
Use the command so that the broker also starts immediately after a restart Code 3.
sudo systemctl enable mosquitto.service
Code 3: Start the mosquitto service when the Pi is restarted
Since the daemon may not yet run on the Pi, start it with the command Code 4.
sudo mosquitto -d
Code 4: start mosquitto
From this point you have set up a local, unencrypted MQTT broker in your home network. If you have not set up port forwarding to the Raspberry Pi via your router, this cannot be reached on the Internet. But it should be enough for private use. So that the broker remains accessible from the Pi after a restart, you should assign the Raspberry Pi a fixed IP address via the router.
Send and receive data to the broker
Now it's time to put theory into practice and send messages to the broker and also receive messages from the broker. For this the terminal commands "mosquitto_sub" for the subscribe function and "mosquitto_pub" for the publish message function are needed.
The terminal command "mosquitto_sub" will start, and the most important commands are explained below. If you want to go beyond the commands shown here, I recommend the internally supplied Linux manual at this point, which you can call up in the terminal with the command “man mosquitto_sub”, “man mosquitto_pub” or “man mosquitto”. The Linux command man stands for manual, in German manual or manual.
To see in detail what the mosquitto subscriber is processing in the background, and also to receive all messages, use the command Code 5.
mosquitto_sub -d -q 2 -t / #
Code 5: Let Mosquitto subscribers listen for root nodes
The one shown here Code 5 uses the options Table 2.
|-d||Activates debug mode and displays all useful information.|
|-q||Specifies which QoS the subscriber should communicate with the broker. QoS 2 is used in Code 5.|
|-t||Specifies which topic the subscriber is to subscribe to. In Code 5, the subscriber subscribes to all topics.|
Table 2: Options of the mosquitto subscriber
Got the command Code 5 is entered, the output will first indicate the successful connection to the broker, see Figure 4 Mark 1.
Figure 4: Terminal output from moquitto_sub
In marker 1 of Figure 4 you can see that a connection is being established and the remote station, the broker, has accepted the connection. Immediately afterwards, the subscriber transmits which topic is to be subscribed to, here "/ #", and which QoS was requested, here "QoS: 2". Finally, the subscribe request is made using SUBACK, the Subscription Acknowledged, accepted. The connection to the broker is established and the desired topic is subscribed to.
Mark 2 of Figure 4 will appear in your terminal after a while. As described in "How does MQTT work", this is the query as to whether the subscriber is still online. The broker asks whether the subscriber is still online, the so-called Ping Request. Normally, the client sends the corresponding response, the so-called Ping Response, to the Broke.
The next step, now that we have a subscriber, is to use a publisher to send data to the broker. The first message should of course be "Hello world". The publisher sends this small sentence to the topic / test / testpub.
To do this, open a second terminal window and issue the command accordingly Code 6 a .
mosquitto_pub -d -q 2 -t / test / testpub -m "Hello world"
Code 6: mosquitto_pub "Hello world"
The options used for Code 6 are in Table 3 listed.
|-d||Activates debug mode and displays all useful information.|
|-q||Specifies which QoS the publisher should communicate with the broker. QoS 2 is used in Code 6.|
|-t||Specifies the topic to which the data should be sent, here / test / testpub|
|-m||Send the following message, here "Hello world"|
Table 3: Options of the mosquitto publisher
Let's take a look at what data is being transmitted, see Figure 5 Mark 1, sent to the broker. As before, QoS 2 was selected in the options, which is why you can also see the corresponding communication flow.
Figure 5: Terminal output from moquitto_pub and mosquitto_sub
Immediately afterwards the broker sends the new message to the subscriber, see Figure 5 Mark 2, because he has subscribed to all topics. On the upper edge of marker 2 of Figure 5 you can see that new data has been received in the / test / testpub topic. The QoS 2 handshake is displayed directly behind it, with the message "Hello world" received.
You have now learned the basics of MQTT and the technical background to MQTT. Feel free to play with the commands once and send data to the broker and, if necessary, open other terminals that subscribe to other topics.
In the next part of the blog series you will connect an Arduino microcontroller or ESP32 to the MQTT and send data, and the remote station will react accordingly.
This and other projects can be found on GitHub at https://github.com/M3taKn1ght/Blog-Repo