Technical note – MIRO Automatic FTP Uploads

This guide introduces how to setup an FTP system with automatic data uploads for the MIRO Power Quality Logger and Analyser.

Introduction

Purpose

The FTP upload option provides automatic data collection without needing an application which periodically polls data to manage it.

This has numerous advantages:

• Unsolicited reporting instead of polling allows for the device (MIRO) to initiate all connections, removing the need for a static IP for each unit.
• Compatible with all SIMs – most public internet SIMs provide a private IP address behind a router/firewall (ie. carrier-grade NAT), effectively blocking all incoming connections. Remember that cellular data services are geared up for smartphones and tablets; machine-to-machine services are a niche application.
• Only the server needs to be static and externally accessible – this is rarely an issue with a wired connection.
• More secure than a polled solution with a public static IP.
• Private IP SIMs can still be used, as long as the server is located within the same private network.
• Many different server options are available across multiple operating systems rather than specifically requiring a Windows PC.
• Third party FTP hosting solutions can be used
• Third party services to receive files via FTP and transfer to another service such as Dropbox are available.

Gapless logging

FTP uploads, like CITRUS downloads, will not interrupt measurement or logging. Data is cleared on successful uploads, and only the data that was for example, and then other PCs may access the files directly through a mapped network drive without needing an FTP client.ou will need either a static IP in your office (usually standard with any wired business internet service) or a dynamic DNS service. A static IP is strongly recommended because this is the simplest and most reliable option.

You will need either a static IP in your office (usually standard with any wired business internet service) or a dynamic DNS service. A static IP is strongly recommended because this is the simplest and most reliable option.

Contact your ISP for more information.

An FTP server is included in Windows Server 2008 and later, but it can be difficult to set up. A popular server that works on any version of Windows is FileZilla: https://filezilla-project.org/

Documentation is available here: https://wiki.filezilla-project.org/Documentation

The server should be set up in “passive” mode. This is usually the default.

Port forwarding will need to be set up on your router. The router’s manual should explain how to do this. Both the primary port (21 by default) and the range of passive-mode ports (eg. ports 5000 to 5100) will both need to be forwarded.

Miro Setup

CITRUS v1.0.4 or later is required, and MIRO firmware v1.08 or later. (Updating to the latest versions from our website is always recommended.)

Logging – Data Volume

The volume of the logged data should be considered carefully as cellular data can be quite limited and/or expensive. An estimated data volume per day and month is provided in the configuration form to help manage this, but the actual amount of data could be larger or smaller depending particularly on how many event captures are taken.

The MIRO does not use segmented memory, and so there is no upper limit to how many waveform/RMS captures can be stored, though (as of firmware 1.08) there is a cap of 300 captures per day. This value may be altered or made configurable in future versions.

For long term installations, it is recommended to compare the size of the first few uploaded files against the estimates. If more events than expected were captured or if the file size is otherwise much larger, consider the following: automatic filename to the end. The name will be the serial number followed by a date and time. Example: 15749004_2016-09-18_04-01-04.miro.

It is recommended to use only letters, numbers, underscores (_) and dashes (-) in the filename.

If files from different units should go in different folders, then either set up a separate FTP username for each, and configure a separate folder in the server, or use the same account but specify a path. Folders will need to be created manually; the MIRO will expect the path to exist already.

Make sure to use forward slashes (/), as this is what FTP expects. This differs from the backslash (\) typically used by Windows.

As an example, the server may be configured with a single username tied to the folder “c:\ftproot”. If three units (serial numbers 1111, 1112, 1113) are set up as follows:

• unit_one/Miro_
• unit_two/Miro_
• unit_three/Miro_

Then the resulting upload paths will be:

• c:\ftproot\unit_one\Miro_1111_2016-09-18_04-00-37.miro
• c:\ftproot\unit_two\Miro_1112_2016-09-18_04-02-20.miro
• c:\ftproot\unit_three\Miro_1113_2016-09-18_04-01-12.miro

Initial Upload

On power up or shortly after FTP uploading is enabled, the MIRO will perform an upload to verify that the connection is working. The Comms page on the LCD can be used to verify that the upload worked if access to the server is not immediately available.

Upload Trigger

Two conditions are currently offered.

• Upload daily or weekly
• Upload when size reaches 20MB or 100MB

Both options may be selected at the same time.

Daily uploads (as of CITRUS 1.1.6) occur at 18:00 UTC time. This will be 4:00 or 5:00 Sydney time depending on daylight savings. Weekly uploads occur early Monday morning at this time.

Remote management and Time Synchronisation

The following features all require firmware v2.13 or later and Citrus v1.1.6 or later.

Configuration

The settings can be found on the FTP upload page:

Firmware Update

If enabled, the Miro will check for firmware updates after a successful upload. It will check for a firmware file with a name similar to the uploaded data file, based on the configured prefix and serial number. For the previous example “unit one”, where the uploaded file path was:

• c:\ftproot\unit_one\Miro_1111_2016-09-18_04-00-37.miro

The firmware update filename would be:

• c:\ftproot\unit_one\Miro_1111.fw

This ensures that only the correct unit is updated.

After downloading and verifying the firmware file, the Miro will delete the file on the server and then reboot to apply the new firmware. As starting up automatically triggers an upload, confirmation is quickly provided that the firmware version was updated.

In short, the update can be performed by taking the latest firmware from our website, uploading it to the FTP server and then moving/renaming it as required, then simply waiting for the upgrade to take place.

Configuration Update

Configuration can be updated in a similar manner. If FTP configuration updates are enabled, the Miro will check after a successful upload for a configuration file with a particular name. Continuing the above example, the file path for “unit one” would be:

• c:\ftproot\unit_one\Miro_1111.miroremoteconfig

actually uploaded is cleared. Any data logged during the upload is maintained, and will be sent during the next upload.

Files from the same MIRO can be joined in CITRUS to produce graphs of weeks or months of data with no gaps.

Connecting with CITRUS

If a static IP is used, it is still possible to connect directly with CITRUS and download even if FTP uploading is enabled.

Server Setup

Server setups differ depending on whether public (internet) or private SIMs are used and (for the public case) on whether the server should be hosted externally or internally.

Private Network

In this case, the server will need to be located within the private network and directly accessible (by hostname or IP) from the remote units.

The files will then be accessible either by connecting to the server via an FTP client (eg. FileZilla Client) or through a mapped network drive.

Your IT department will need to set this up.

Third-party hosted (recommended)

FTP hosting is widely available with different options for storage, monthly data caps, user accounts etc.

Data can then be downloaded from the server by connecting to the server with any FTP client (such as FileZilla client).

Contact us if you are unsure which provider to use. We can help select a provider/plan and set it up on your behalf.

Self-hosted (advanced)

This option will take longer to set up but will be cheaper and may be easier to use. The FTP server could be configured to place files in a shared folder,

• Set the adjustable interval to 60 seconds or more, or disable all except IEC 10 minute and 2 hour intervals.
• Enable only the first 25 or 50 harmonics unless problems at higher harmonics are suspected.
• Disable sliding reference triggering from current:
• All channels including current are always captured regardless of the trigger source
• Large jumps in current usually have a corresponding voltage drop
• Raise the sliding reference threshold(s).
• Raise the transient threshold. In some installations, there will be large numbers of relatively minor events such as that shown in figure 1, which may not be of interest.

• Disable mains signalling captures and use the maximum-detection only.

Server Details

Regardless of how the server is set up, there are four pieces of information which you need:

• Server hostname or IP. Examples:
  • ftp.example.org (publicly accessible hostname)
  • 203.0.113.20 (publicly accessible IP)
  • 172.16.100.1 (private IP)
• Server port, such as 21
• Username
• Password

A path and/or prefix is optional. The MIRO will take the value entered in the text box and attach an A random delay up to 5 minutes is added to prevent multiple units all trying to log in to the server at the same moment.

This time was chosen as cellular network load is usually very light at this time. The exact time is arbitrary as the upload process does not leave any gaps in the data.

Example

In this example, the MIRO will connect to a server we have set up in a workstation in our office, and upload files into the folder test­_daily­­_upload daily or earlier if data reaches 20MB:

After running for a few days, these files were available on the server:

Retries

If an upload fails, the Miro will attempt the upload again after a delay. There is a maximum of five retries. The delay starts at one minute and doubles each try (“exponential backoff”). After five failed retries, the upload is cancelled and the Miro returns to waiting for an upload trigger condition.

(If the size trigger is used, then it will re-trigger immediately, as the memory will not have been cleared, and the process will loop until the server issue or communications issue is resolved.)

The Comms page on the LCD indicates if the device is currently waiting to retry or if the process failed.

1 . After downloading and validating this file, the Miro will apply the configuration and then delete it from the server. As the Miro was designed to allow configuration at any time, it will keep operating as usual with no interruption to logging.

The file must be in “remote configuration” format. This format contains all the same information as a normal configuration file, except that it requires the user to explicitly authorise it for automatic remote application.

To produce the file, the user should load the configuration from the most recent upload by opening offline configuration and loading the most recent data file, making any necessary changes, then pressing Save To File. If the remote configuration setting is enabled, Citrus will ask the user if the file should be saved in remote-configuration format.

Save this file with the appropriate name, then upload it to the FTP server. Unlike the firmware upgrade, configuration updates won’t trigger an upload, but this isn’t necessary as the deletion takes place after successfully updating. Therefore, seeing the file be deleted is sufficient to confirm the update.

Time Synchronisation – NTP

While the typical clock drift is only one second per week, regular time synchronisation is recommended. NTP can be used to do this automatically, and like FTP it is triggered from the device side and therefore works with all SIM cards and network setups.

Enable Sync Time under the FTP menu. If a particular NTP server is preferred, or if using a private network with no internet access, enter the hostname or IP in the box provided. Otherwise we recommend the following server: pool.ntp.org. This is a publicly available server pool which will automatically redirect the request to an appropriate nearby server.

If NTP is enabled but no server is provided, the Miro will attempt to use the hostname/IP set as the FTP server for NTP as well.

Time sync is triggered automatically after a successful upload. The device will send three requests, take the median offset to the server (accounting for latency) and adjust the clock.

With daily uploads, this will ensure the clock is always within one second of real time.