Viewing file:  mod_redis.html (18.9 KB) -rw-r--r--
Select action/file-type:
 (+) | (+) |  (+) | Code (+) | Session (+) |  (+) | SDB (+) |  (+) |  (+) |  (+) |  (+) |  (+) |
ProFTPD module mod_redis
ProFTPD module mod_redis
The mod_redis module enables ProFTPD support for caching data in
Redis servers, using the
hiredis client library.
Directives
Syntax: RedisEngine on|off
Default: RedisEngine off
Context: server config, <VirtualHost>, <Global>
Module: mod_redis
Compatibility: 1.3.6rc5 and later
The RedisEngine directive enables or disables the
mod_redis module, and thus the configuration of Redis support for
the proftpd daemon.
Syntax: RedisLog path|"none"
Default: None
Context: server config, <VirtualHost>, <Global>
Module: mod_redis
Compatibility: 1.3.6rc5 and later
The RedisLog directive is used to specify a log file for
mod_redis's reporting on a per-server basis. The
file parameter given must be the full path to the file to use for
logging.
Note that this path must not be to a world-writable directory and,
unless AllowLogSymlinks is explicitly set to on
(generally a bad idea), the path must not be a symbolic link.
Syntax: RedisLogOnCommand "none"|commands format-name [key]
Default: None
Context: server config, <VirtualHost>, <Global>, <Anonymous>, <Directory>
Module: mod_redis
Compatibility: 1.3.6rc5 and later
The RedisLogOnCommand directive configures the use of Redis for
logging. Whenever one of the comma-separated list of commands
occurs, mod_redis will compose a JSON object, using the
LogFormat named by
format-name as a template for the fields to include in the
JSON object. The JSON object of that event will then be appended to a list
stored in Redis, using format-name as the default key name. Multiple
RedisLogOnCommand directives can be used, for different log formats
for different events.
The optional key parameter, if present, specifies the value to use
as the Redis key. This key parameter supports all of the
LogFormat variables, thus you can use e.g.:
RedisLogOnCommand RETR,STOR xferlog %m:ftpxfer:%u
More on the use of Redis logging, including a table showing how
LogFormat variables are mapped to JSON object keys can be found
here.
Example:
LogFormat file-transfers "%h %l %u %t \"%r\" %s %b"
# Only log to Redis for uploads in this directory
RedisLogOnCommand APPE,STOR,STOU file-transfers
# Only log to Redis for downloads from this directory
RedisLogOnCommand RETR file-transfers
# ...but prevent Redis logging in this subdirectory
RedisLogOnCommand none
Note that RedisLogOnCommand does not currently
support the logging classes that the ExtendedLog directive
supports.
Syntax: RedisLogOnEvent "none"|events format-name [key]
Default: None
Context: server config, <VirtualHost>, <Global>, <Anonymous>, <Directory>
Module: mod_redis
Compatibility: 1.3.7rc1 and later
The RedisLogOnEvent directive configures the use of Redis for
logging. Whenever one of the comma-separated list of events
occurs, mod_redis will compose a JSON object, using the
LogFormat named by
format-name as a template for the fields to include in the
JSON object. The JSON object of that event will then be appended to a list
stored in Redis, using format-name as the default key name. Multiple
RedisLogOnEvent directives can be used, for different log formats
for different events.
The optional key parameter, if present, specifies the value to use
as the Redis key. This key parameter supports all of the
LogFormat variables, thus you can use e.g.:
RedisLogOnEvent READ ftp.download.%u.%m:%f
More on the use of Redis logging, including a table showing how
LogFormat variables are mapped to JSON object keys can be found
here.
Example:
LogFormat sessions "%{iso8601} %a"
RedisLogOnEvent CONNECT,DISCONNECT sessions
In addition to specific FTP commands, the events list can specify
"ALL", for logging on all commands. Or it can include the
"CONNECT" and "DISCONNECT" events, which can be useful for logging the
start and end times of a session. Note that
RedisLogOnEvent does support the logging classes
that the ExtendedLog directive supports.
Syntax: RedisOptions opt1 ...
Default: None
Context: server config, <VirtualHost>, <Global>
Module: mod_redis
Compatibility: 1.3.7rc1 and later
The RedisOptions directive is used to configure various optional
behavior of mod_redis.
Example:
RedisOptions NoReconnect
The currently implemented options are:
Syntax: RedisSentinel host1 ... ["master" name]
Default: None
Context: server config, <VirtualHost>, <Global>
Module: mod_redis
Compatibility: 1.3.7rc1 and later
The RedisSentinel directive is used to configure a list of IP
addresses/ports of Redis Sentinels that the mod_redis module is
to use, for discovering the location of named master database. For
example:
# Configure two Sentinels; the first master discovered will be used
RedisSentinel 1.2.3.4:26379 5.6.7.8:36379
# Configure two Sentinels on the default Sentinel port, and
# look for the location of the master named "proftpd".
RedisSentinel 1.2.3.4 5.6.7.8 master proftpd
# Configure three Sentinels including an IPv6 address, with explicit ports,
# and look for the location of the "proftpd" master.
RedisSentinel redis1.example.com:26379 1.2.3.4:26379 [::1]:26379 master proftpd
The RedisSentinel directive can be used instead of the
RedisServer directive, for discovering the Redis server to use.
However, if your Redis server requires authentication, or
supports multiple databases, then you will need to use
RedisServer as well:
# Use this to configure our password and database
RedisServer 127.0.0.1:6379 redisr0cks 2
# And use Sentinels to discover the true address/port
RedisSentine 1.2.3.4 5.6.7.8 9.10.11.12
Syntax: RedisServer server [password] [db-index]
Default: None
Context: server config, <VirtualHost>, <Global>
Module: mod_redis
Compatibility: 1.3.6rc5 and later
The RedisServer directive is used to configure the IP address/port
of the Redis server that the mod_redis module is to use. For
example:
RedisServer 1.2.3.4:6379
or, for an IPv6 address, make sure the IPv6 address is enclosed in square
brackets:
RedisServer [::ffff:1.2.3.4]:6379
Alternatively, you can configure a Unix domain socket path using e.g.:
RedisServer /var/run/redis.sock
An optional password parameter can be provided, for Redis servers
which are password protected.
As of ProFTPD 1.3.7rc1, the optional db-index parameter can be
provided, for selecting the server-side Redis database by index:
RedisServer 1.2.3.4:6379 passwd 2
Connecting to a Redis server without password authentication but still
selecting the database would be done using the empty string for the password:
RedisServer 1.2.3.4:6379 "" 2
Syntax: RedisTimeouts connect-millis io-millis
Default: RedisTimeouts 500 500
Context: server config, <VirtualHost>, <Global>
Module: mod_redis
Compatibility: 1.3.6rc5 and later
The RedisTimeouts directive configures timeouts to be used
when communicating with the Redis server. The connect-millis
parameter specifies a timeout, in milliseconds, to use when first
connecting to the Redis server. The io-millis parameter specifies
a timeout, in milliseconds, to use both when sending commands to Redis, and
when reading responses.
The default is 500 milliseconds for both timeouts:
RedisTimeouts 500 500
The mod_redis module is distributed with ProFTPD. To enable
support and use of the Redis protocol in your proftpd daemon,
use the --enable-redis configure option:
$ ./configure --enable-redis ...
$ make
$ make install
This option causes the mod_redis module to be compiled into
proftpd.
You may also need to tell configure how to find the
hiredis header and library files:
$ ./configure --enable-redis \
--with-includes=/path/to/hiredis/include \
--with-libraries=/path/to/hiredis/lib
Configuring Redis for use by other modules, e.g. mod_ban
or mod_tls_redis:
<IfModule mod_redis.c>
RedisEngine on
RedisLog /var/log/ftpd/redis.log
RedisServer 127.0.0.1:6379
</IfModule>
This example shows the use of Redis logging for all commands:
<IfModule mod_redis.c>
RedisEngine on
RedisLog /var/log/ftpd/redis.log
RedisServer 127.0.0.1:6379
LogFormat redis "%h %l %u %t \"%r\" %s %b"
RedisLogOnCommand ALL redis
</IfModule>
Redis Logging
When using Redis logging, the following table shows how mod_redis
converts a LogFormat variable into the key names in the JSON
logging objects:
LogFormat Variable |
Key |
%A |
anon_password |
%a |
remote_ip |
%b |
bytes_sent |
%c |
connection_class |
%D |
dir_path |
%d |
dir_name |
%E |
session_end_reason |
%{epoch} |
Unix timestamp, in seconds since Jan 1 1970. |
%{name}e |
ENV:name |
%F |
transfer_path |
%f |
file |
%{file-modified} |
file_modified |
%g |
group |
%{gid} |
gid |
%H |
server_ip |
%h |
remote_dns |
%I |
session_bytes_rcvd |
%{iso8601} |
timestamp |
%J |
command_params |
%L |
local_ip |
%l |
identd_user |
%m |
command |
%{microsecs} |
microsecs |
%{millisecs} |
millisecs |
%{note:name} |
NOTE:name |
%O |
session_bytes_sent |
%P |
pid |
%p |
local_port |
%{protocol} |
protocol |
%r |
raw_command |
%S |
response_msg |
%s |
response_code |
%T |
transfer_secs |
%t |
local_time |
%{transfer-failure} |
transfer_failure |
%{transfer-status} |
transfer_status |
%U |
original_user |
%u |
user |
%{uid} |
uid |
%V |
server_dns |
%v |
server_name |
%{version} |
server_version |
%w |
rename_from |
In addition to the standard LogFormat variables, the
mod_redis module also adds a "connecting" key for events
generated when a client first connects, and a "disconnecting" key for events
generated when a client disconnects. These keys can be used for determining
the start/finish events for a given session.
Here is an example of the JSON-formatted records generated, using the above
example configuration:
{"connecting":true,"timestamp":"2013-08-21 23:08:22,171"}
{"command":"USER","timestamp":"2013-08-21 23:08:22,278"}
{"user":"proftpd","command":"PASS","timestamp":"2013-08-21 23:08:22,305"}
{"user":"proftpd","command":"PASV","timestamp":"2013-08-21 23:08:22,317"}
{"user":"proftpd","command":"LIST","bytes_sent":432,"transfer_secs":4.211,"timestamp":"2013-08-21 23:08:22,329"}
{"user":"proftpd","command":"QUIT","timestamp":"2013-08-21 23:08:22,336"}
{"disconnecting":true,"user":"proftpd","timestamp":"2013-08-21 23:08:22,348"}
Notice that for a given event, not all of the LogFormat
variables are filled in. If mod_redis determines that a given
LogFormat variable has no value for the logged event, it will
simply omit that variable from the JSON object.
Another thing to notice is that the generated JSON object ignores the textual
delimiters configured by the LogFormat directive; all that
matters are the LogFormat variables which appear in the directive.
Frequently Asked Questions
Question: How can I convert this SQL logging into the
equivalent Redis logging?
SQLNamedQuery upload FREEFORM "INSERT INTO ftplogs ('userid', 'server_ip', 'transfer_date', 'operation', 'protocol', 'client_ip', 'transfer_time', 'bytes_transfer', 'file_hash_type', 'file_hash', 'file_path', 'transfer_status') VALUES ('%u', '%H', NOW(), '%r', '%{protocol}', '%a', '%T', '%b', '%{note:mod_digest.algo}', '%{note:mod_digest.digest}', '%f', '%{transfer-status}')"
SQLLog STOR upload
Answer: Since the JSON object key names are hardcoded
in mod_redis, converting the above SQLNamedQuery
into a suitable/matching LogFormat is the necessary step. Thus
for example it might become:
LogFormat upload "%u %H %{YYYY-MM-DD HH:MM:SS}t %r %{protocol} %a %T %b %{note:mod_digest.algo} %{note:mod_digest.digest} %f %{transfer-status}"
RedisLogOnCommand STOR upload
Note that LogFormat does not provide a NOW()
function, unlike many SQL databases, thus the %t variable is
needed to provide/fill in that timestamp.
© Copyright 2017 The ProFTPD Project
All Rights Reserved
|