Caching of ondemand pools and status checks (#19)

* [add] zero exit code on success

* [add] cache for ondemand pm and check status of pools during discovery

* [fix] improved detection of some commands

* [add] info about ondemand pools, update syntax of discovery script in samples

README.md

@@ -11,7 +11,7 @@
 - Supported types of PHP [process manager](https://www.php.net/manual/en/install.fpm.configuration.php#pm):
 	- [x] dynamic
 	- [x] static
-	- [ ] ondemand - has some problems, because of its working logic, see issue [#11](https://github.com/rvalitov/zabbix-php-fpm/issues/11) 
+	- [x] ondemand. Such pools are invisible (undiscoverable) if they are not active because of their nature, i.e. when no PHP-FPM processes related to the pools spawned during the discovery process of Zabbix agent. After a pool has been discovered for the first time, it becomes permanently visible for Zabbix. Regular checks performed by Zabbix agent require at least one active PHP-FPM process that can report the status, and if such process does not exist, then it will be spawned. As a result, Zabbix agent will always report that there's at least one active PHP-FPM process for the pool. Besides, there's a chance that such behaviour may have a negative impact on the pool's performance and you may consider changing to another type of process manager, for example, dynamic.   
 - Supports multiple PHP versions, i.e. you can use PHP 7.2 and PHP 7.3 on the same server and we will detect them all
 - Easy configuration
 - Supports [ISPConfig](https://www.ispconfig.org/)
@@ -281,13 +281,13 @@ Here we specified `zabbix` as the user under which the Zabbix Agent is run. This
 Now edit the file `userparameter_php_fpm.conf`. Find the line:
 
 ```
-UserParameter=php-fpm.discover,/etc/zabbix/zabbix_php_fpm_discovery.sh
+UserParameter=php-fpm.discover[*],/etc/zabbix/zabbix_php_fpm_discovery.sh $1
 ```
 
 Add `sudo` there, so the line should be:
 
 ```
-UserParameter=php-fpm.discover,sudo /etc/zabbix/zabbix_php_fpm_discovery.sh
+UserParameter=php-fpm.discover[*],sudo /etc/zabbix/zabbix_php_fpm_discovery.sh $1
 ```
 
 That's all. 
@@ -422,10 +422,10 @@ The setup is finished, just wait a couple of minutes till Zabbix discovers all y
 
 # Testing and Troubleshooting
 ## Check auto discovery
-First test that auto discovery of PHP-FPM pools works on your machine. Run the following command:
+First test that auto discovery of PHP-FPM pools works on your machine. Run the following command (replace `POOL_PATH` with the status path of PHP-FPM that you set in [`pm.status_path`](https://github.com/rvalitov/zabbix-php-fpm#16-adjust-php-fpm-pools-configuration), the default value is `/php-fpm-status`):
 
 ```console
-root@server:/etc/zabbix#bash /etc/zabbix/zabbix_php_fpm_discovery.sh
+root@server:/etc/zabbix#bash /etc/zabbix/zabbix_php_fpm_discovery.sh POOL_PATH
 ```
 **Important:** please make sure that you use `bash` in the command above, not `sh` or other alternatives, otherwise you may get a script syntax error message.
 
@@ -452,7 +452,7 @@ The output should be a valid JSON with a list of pools and their sockets, someth
 
 For further investigation you can run the script above with `debug` option to get more details, example:
 ```console
-root@server:/etc/zabbix#bash /etc/zabbix/zabbix_php_fpm_discovery.sh debug
+root@server:/etc/zabbix#bash /etc/zabbix/zabbix_php_fpm_discovery.sh POOL_PATH debug
 Debug mode enabled
 Success: found socket /var/lib/php7.3-fpm/web1.sock for pool web1, raw process info: php-fpm7. 5094 web1 11u unix 0x00000000dd9ea858 0t0 104495372 /var/lib/php7.3-fpm/web1.sock type=STREAM
 Success: found socket /var/lib/php7.3-fpm/web4.sock for pool web4, raw process info: php-fpm7. 5096 web4 11u unix 0x00000000562748dd 0t0 104495374 /var/lib/php7.3-fpm/web4.sock type=STREAM
@@ -501,7 +501,7 @@ apt-get install zabbix-get
 Example how to discover PHP-FPM pools:
 
 ```console
-root@server:/# zabbix_get -s 127.0.0.1 -p 10050 -k php-fpm.discover
+root@server:/# zabbix_get -s 127.0.0.1 -p 10050 -k php-fpm.discover POOL_PATH
 {"data":[{"{#POOLNAME}":"www","{#POOLSOCKET}":"/run/php/php7.3-fpm.sock"},{"{#POOLNAME}":"www2","{#POOLSOCKET}":"localhost:9001"}]}
 ```
 
@@ -511,7 +511,7 @@ To get status of the required pool, use the following command:
 zabbix_get -s 127.0.0.1 -p 10050 -k php-fpm.discover.status[POOL_URL,POOL_PATH]
 ```
 
-In the above example we use the following values:
+In the above examples we use the following values:
 
 - `127.0.0.1` is the IP address of the host where the Zabbix Agent is installed and where the PHP-FPM is running
 - `10050` is the port of the Zabbix Agent

zabbix/userparameter_php_fpm.conf

@@ -1,2 +1,2 @@
-UserParameter=php-fpm.discover,/etc/zabbix/zabbix_php_fpm_discovery.sh
+UserParameter=php-fpm.discover[*],/etc/zabbix/zabbix_php_fpm_discovery.sh $1
-UserParameter=php-fpm.status[*],/etc/zabbix/zabbix_php_fpm_status.sh $1 $2
+UserParameter=php-fpm.status[*],/etc/zabbix/zabbix_php_fpm_status.sh $1 $2

zabbix/zabbix_php_fpm_discovery.sh

@@ -10,57 +10,209 @@ S_SORT=`type -P sort`
 S_HEAD=`type -P head`
 S_LSOF=`type -P lsof`
 S_JQ=`type -P jq`
+S_DIRNAME=`type -P dirname`
+S_CAT=`type -P cat`
+S_BASH=`type -P bash`
+S_ECHO=`type -P echo`
+S_PRINTF=`type -P printf`
 
 if [[ ! -f $S_PS ]]; then
-	echo "Utility 'ps' not found. Please, install it first."
+	${S_ECHO} "Utility 'ps' not found. Please, install it first."
 	exit 1
 fi
 if [[ ! -f $S_GREP ]]; then
-	echo "Utility 'grep' not found. Please, install it first."
+	${S_ECHO} "Utility 'grep' not found. Please, install it first."
 	exit 1
 fi
 if [[ ! -f $S_AWK ]]; then
-	echo "Utility 'awk' not found. Please, install it first."
+	${S_ECHO} "Utility 'awk' not found. Please, install it first."
 	exit 1
 fi
 if [[ ! -f $S_SORT ]]; then
-	echo "Utility 'sort' not found. Please, install it first."
+	${S_ECHO} "Utility 'sort' not found. Please, install it first."
 	exit 1
 fi
 if [[ ! -f $S_HEAD ]]; then
-	echo "Utility 'head' not found. Please, install it first."
+	${S_ECHO} "Utility 'head' not found. Please, install it first."
 	exit 1
 fi
 if [[ ! -f $S_LSOF ]]; then
-	echo "Utility 'lsof' not found. Please, install it first."
+	${S_ECHO} "Utility 'lsof' not found. Please, install it first."
 	exit 1
 fi
 if [[ ! -f $S_JQ ]]; then
-	echo "Utility 'jq' not found. Please, install it first."
+	${S_ECHO} "Utility 'jq' not found. Please, install it first."
+	exit 1
+fi
+if [[ ! -f ${S_DIRNAME} ]]; then
+	${S_ECHO} "Utility 'dirname' not found. Please, install it first."
+	exit 1
+fi
+if [[ ! -f ${S_CAT} ]]; then
+	${S_ECHO} "Utility 'cat' not found. Please, install it first."
+	exit 1
+fi
+if [[ ! -f ${S_BASH} ]]; then
+	${S_ECHO} "Utility 'bash' not found. Please, install it first."
+	exit 1
+fi
+if [[ ! -f ${S_PRINTF} ]]; then
+	${S_ECHO} "Utility 'printf' not found. Please, install it first."
 	exit 1
 fi
 
+STATUS_PATH="/php-fpm-status"
 DEBUG_MODE=""
-if [[ ! -z $1 ]] && [[ $1 == "debug" ]]; then
-    DEBUG_MODE="1"
-    echo "Debug mode enabled"
-fi
 
 # Prints a string on screen. Works only if debug mode is enabled.
 function PrintDebug(){
 	if [[ ! -z $DEBUG_MODE ]] && [[ ! -z $1 ]]; then
-	    echo $1
+	    ${S_ECHO} $1
 	fi
 }
 
+# Encodes input data to JSON and saves it to result string
+# Input arguments:
+# - pool name
+# - pool socket
+# Function returns 1 if all OK, and 0 otherwise.
+function EncodeToJson(){
+    POOL_NAME=$1
+    POOL_SOCKET=$2
+    if [[ -z ${POOL_NAME} ]] || [[ -z ${POOL_SOCKET} ]]; then
+        return 0
+    fi
+
+    JSON_POOL=`${S_ECHO} -n "$POOL_NAME" | ${S_JQ} -aR .`
+    JSON_SOCKET=`${S_ECHO} -n "$POOL_SOCKET" | ${S_JQ} -aR .`
+    if [[ ${POOL_FIRST} == 1 ]]; then
+        RESULT_DATA="$RESULT_DATA,"
+    fi
+    RESULT_DATA="$RESULT_DATA{\"{#POOLNAME}\":$JSON_POOL,\"{#POOLSOCKET}\":$JSON_SOCKET}"
+    POOL_FIRST=1
+    return 1
+}
+
+# Checks if selected pool is in cache.
+# Input arguments:
+# - pool name
+# - pool socket
+# Function returns 1 if the pool is in cache, and 0 otherwise.
+function IsInCache(){
+    SEARCH_NAME=$1
+    SEARCH_SOCKET=$2
+    if [[ -z ${SEARCH_NAME} ]] || [[ -z ${SEARCH_SOCKET} ]]; then
+        return 0
+    fi
+    for CACHE_ITEM in "${NEW_CACHE[@]}"
+    do
+        ITEM_NAME=`${S_ECHO} "$CACHE_ITEM" | ${S_AWK} '{print $1}'`
+        ITEM_SOCKET=`${S_ECHO} "$CACHE_ITEM" | ${S_AWK} '{print $2}'`
+        if [[ ${ITEM_NAME} == ${SEARCH_NAME} ]] && [[ ${ITEM_SOCKET} == ${SEARCH_SOCKET} ]]; then
+            return 1
+        fi
+    done
+    return 0
+}
+
+# Validates the specified pool by getting its status and working with cache.
+# Pass two arguments: pool name and pool socket
+# Function returns:
+# 0 if the pool is invalid
+# 1 if the pool is OK and is ondemand and is not in cache
+# 2 if the pool is OK and is ondemand and is in cache
+# 3 if the pool is OK and is not ondemand and is not in cache
+function ProcessPool(){
+    POOL_NAME=$1
+    POOL_SOCKET=$2
+    if [[ -z ${POOL_NAME} ]] || [[ -z ${POOL_SOCKET} ]]; then
+        return 0
+    fi
+
+    IsInCache ${POOL_NAME} ${POOL_SOCKET}
+    FOUND=$?
+    if [[ ${FOUND} == 1 ]]; then
+        return 2
+    fi
+
+    STATUS_JSON=`${S_BASH} ${STATUS_SCRIPT} ${POOL_SOCKET} ${STATUS_PATH}`
+    EXIT_CODE=$?
+    if [[ ${EXIT_CODE} == 0 ]]; then
+        # The exit code is OK, let's check the JSON data
+        # JSON data example:
+        # {"pool":"www2","process manager":"ondemand","start time":1578181845,"start since":117,"accepted conn":3,"listen queue":0,"max listen queue":0,"listen queue len":0,"idle processes":0,"active processes":1,"total processes":1,"max active processes":1,"max children reached":0,"slow requests":0}
+        # We use basic regular expression here, i.e. we need to use \+ and not escape { and }
+        if [[ ! -z `${S_ECHO} ${STATUS_JSON} | ${S_GREP} -G '^{.*\"pool\":\".\+\".*,\"process manager\":\".\+\".*}$'` ]]; then
+            PrintDebug "Status data for pool $POOL_NAME, socket $POOL_SOCKET, status path $STATUS_PATH is valid"
+            # Checking if we have ondemand pool
+            if [[ ! -z `${S_ECHO} ${STATUS_JSON} | ${S_GREP} -F '"process manager":"ondemand"'` ]]; then
+                PrintDebug "Detected pool's process manager is ondemand, it needs to be cached"
+                NEW_CACHE+=("$POOL_NAME $POOL_SOCKET")
+                return 1
+            fi
+            PrintDebug "Detected pool's process manager is NOT ondemand, it will not be cached"
+            return 3
+        fi
+
+        PrintDebug "Failed to validate status data for pool $POOL_NAME, socket $POOL_SOCKET, status path $STATUS_PATH"
+        if [[ ! -z ${STATUS_JSON} ]]; then
+            PrintDebug "Status script returned: $STATUS_JSON"
+        fi
+        return 0
+    fi
+    PrintDebug "Failed to get status for pool $POOL_NAME, socket $POOL_SOCKET, status path $STATUS_PATH"
+    if [[ ! -z ${STATUS_JSON} ]]; then
+        PrintDebug "Status script returned: $STATUS_JSON"
+    fi
+    return 0
+}
+
+for ARG in "$@"; do
+    if [[ ${ARG} == "debug" ]]; then
+        DEBUG_MODE="1"
+        ${S_ECHO} "Debug mode enabled"
+    elif [[ ${ARG} == /* ]]; then
+        STATUS_PATH=${ARG}
+        PrintDebug "Argument $ARG is interpreted as status path"
+    else
+        PrintDebug "Argument $ARG is unknown and skipped"
+    fi
+done
+PrintDebug "Status path to be used: $STATUS_PATH"
+
+LOCAL_DIR=`${S_DIRNAME} $0`
+CACHE_FILE="$LOCAL_DIR/php_fpm.cache"
+STATUS_SCRIPT="$LOCAL_DIR/zabbix_php_fpm_status.sh"
+PrintDebug "Local directory is $LOCAL_DIR"
+if [[ ! -f ${STATUS_SCRIPT} ]]; then
+    ${S_ECHO} "Helper script $STATUS_SCRIPT not found"
+    exit 1
+fi
+if [[ ! -r ${STATUS_SCRIPT} ]]; then
+    ${S_ECHO} "Helper script $STATUS_SCRIPT is not readable"
+    exit 1
+fi
+PrintDebug "Helper script $STATUS_SCRIPT is reachable"
+
+# Loading cached data for ondemand pools.
+# The cache file consists of lines, each line contains pool name, then space, then socket (or TCP info)
+CACHE=()
+NEW_CACHE=()
+if [[ -r ${CACHE_FILE} ]]; then
+    PrintDebug "Reading cache file $CACHE_FILE..."
+    mapfile -t CACHE < <( ${S_CAT} ${CACHE_FILE} )
+else
+    PrintDebug "Cache file $CACHE_FILE not found, skipping..."
+fi
+
 mapfile -t PS_LIST < <( $S_PS ax | $S_GREP -F "php-fpm: pool " | $S_GREP -F -v "grep" )
-POOL_LIST=`printf '%s\n' "${PS_LIST[@]}" | $S_AWK '{print $NF}' | $S_SORT -u`
+POOL_LIST=`${S_PRINTF} '%s\n' "${PS_LIST[@]}" | $S_AWK '{print $NF}' | $S_SORT -u`
 POOL_FIRST=0
 #We store the resulting JSON data for Zabbix in the following var:
 RESULT_DATA="{\"data\":["
 while IFS= read -r line
 do
-    POOL_PID=`printf '%s\n' "${PS_LIST[@]}" | $S_GREP -F -w "php-fpm: pool $line" | $S_HEAD -1 | $S_AWK '{print $1}'`
+    POOL_PID=`${S_PRINTF} '%s\n' "${PS_LIST[@]}" | $S_GREP -F -w "php-fpm: pool $line" | $S_HEAD -1 | $S_AWK '{print $1}'`
     if [[ ! -z $POOL_PID ]]; then
         #We search for socket or IP address and port
         #Socket example:
@@ -83,29 +235,43 @@ do
             if [[ ! -z $pool ]]; then
                 if [[ -z $FOUND_POOL ]]; then
                     PrintDebug "Checking process: $pool"
-                    POOL_TYPE=`echo "${pool}" | $S_AWK '{print $5}'`
+                    POOL_TYPE=`${S_ECHO} "${pool}" | $S_AWK '{print $5}'`
-                    POOL_SOCKET=`echo "${pool}" | $S_AWK '{print $9}'`
+                    POOL_SOCKET=`${S_ECHO} "${pool}" | $S_AWK '{print $9}'`
                     if [[ ! -z $POOL_TYPE ]] && [[ ! -z $POOL_SOCKET ]]; then
                         if [[ $POOL_TYPE == "unix" ]]; then
                             #We have a socket here, test if it's actually a socket:
                             if [[ -S $POOL_SOCKET ]]; then
-                                FOUND_POOL="1"
+                                PrintDebug "Found socket $POOL_SOCKET"
-                                PrintDebug "Success: found socket $POOL_SOCKET"
+                                ProcessPool ${line} ${POOL_SOCKET}
+                                POOL_STATUS=$?
+                                if [[ ${POOL_STATUS} > 0 ]]; then
+                                    FOUND_POOL="1"
+                                    PrintDebug "Success: socket $POOL_SOCKET returned valid status data"
+                                else
+                                    PrintDebug "Error: socket $POOL_SOCKET didn't return valid data"
+                                fi
                             else
                                 PrintDebug "Error: specified socket $POOL_SOCKET is not valid"
                             fi
                         elif [[ $POOL_TYPE == "IPv4" ]] || [[ $POOL_TYPE == "IPv6" ]]; then
                             #We have a TCP connection here, check it:
-                            CONNECTION_TYPE=`echo "${pool}" | $S_AWK '{print $8}'`
+                            CONNECTION_TYPE=`${S_ECHO} "${pool}" | $S_AWK '{print $8}'`
                             if [[ $CONNECTION_TYPE == "TCP" ]]; then
                                 #The connection must have state LISTEN:
-                                LISTEN=`echo ${pool} | $S_GREP -F -w "(LISTEN)"`
+                                LISTEN=`${S_ECHO} ${pool} | $S_GREP -F -w "(LISTEN)"`
                                 if [[ ! -z $LISTEN ]]; then
                                     #Check and replace * to localhost if it's found. Asterisk means that the PHP listens on
                                     #all interfaces.
-                                    POOL_SOCKET=`echo -n ${POOL_SOCKET/*:/localhost:}`
+                                    POOL_SOCKET=`${S_ECHO} -n ${POOL_SOCKET/*:/localhost:}`
-                                    FOUND_POOL="1"
+                                    PrintDebug "Found TCP connection $POOL_SOCKET"
-                                    PrintDebug "Success: found TCP connection $POOL_SOCKET"
+                                    ProcessPool ${line} ${POOL_SOCKET}
+                                    POOL_STATUS=$?
+                                    if [[ ${POOL_STATUS} > 0 ]]; then
+                                        FOUND_POOL="1"
+                                        PrintDebug "Success: TCP connection $POOL_SOCKET returned valid status data"
+                                    else
+                                        PrintDebug "Error: TCP connection $POOL_SOCKET didn't return valid data"
+                                    fi
                                 else
                                     PrintDebug "Warning: expected connection state must be LISTEN, but it was not detected"
                                 fi
@@ -126,14 +292,8 @@ do
             fi
         done <<< "$POOL_PARAMS_LIST"
 
-        if [[ ! -z $FOUND_POOL ]]; then
+        if [[ ! -z ${FOUND_POOL} ]]; then
-            JSON_POOL=`echo -n "$line" | $S_JQ -aR .`
+            EncodeToJson ${line} ${POOL_SOCKET}
-            JSON_SOCKET=`echo -n "$POOL_SOCKET" | $S_JQ -aR .`
-            if [[ $POOL_FIRST == 1 ]]; then
-                RESULT_DATA="$RESULT_DATA,"
-            fi
-            RESULT_DATA="$RESULT_DATA{\"{#POOLNAME}\":$JSON_POOL,\"{#POOLSOCKET}\":$JSON_SOCKET}"
-            POOL_FIRST=1
         else
             PrintDebug "Error: failed to discover information for pool $line"
         fi
@@ -141,6 +301,23 @@ do
         PrintDebug "Error: failed to find PID for pool $line"
     fi
 done <<< "$POOL_LIST"
+
+PrintDebug "Processing pools from old cache..."
+for CACHE_ITEM in "${CACHE[@]}"
+do
+    ITEM_NAME=`${S_ECHO} "$CACHE_ITEM" | ${S_AWK} '{print $1}'`
+    ITEM_SOCKET=`${S_ECHO} "$CACHE_ITEM" | ${S_AWK} '{print $2}'`
+    ProcessPool ${ITEM_NAME} ${ITEM_SOCKET}
+    POOL_STATUS=$?
+    if [[ ${POOL_STATUS} == "1" ]]; then
+        # This is a new pool and we must add it
+        EncodeToJson ${ITEM_NAME} ${ITEM_SOCKET}
+    fi
+done
+
+PrintDebug "Saving new cache file $CACHE_FILE..."
+${S_PRINTF} "%s\n" "${NEW_CACHE[@]}" > ${CACHE_FILE}
+
 RESULT_DATA="$RESULT_DATA]}"
 PrintDebug "Resulting JSON data for Zabbix:"
-echo -n $RESULT_DATA
+${S_ECHO} -n $RESULT_DATA

zabbix/zabbix_php_fpm_status.sh

@@ -5,6 +5,7 @@
 
 S_FCGI=`type -P cgi-fcgi`
 S_GREP=`type -P grep`
+S_ECHO=`type -P echo`
 
 if [[ ! -f $S_FCGI ]]; then
 	echo "Utility 'cgi-fcgi' not found. Please, install it first."
@@ -34,4 +35,5 @@ SCRIPT_FILENAME=$POOL_PATH \
 QUERY_STRING=json \
 REQUEST_METHOD=GET \
 $S_FCGI -bind -connect $POOL_URL 2>/dev/null`
-echo "$PHP_STATUS" | $S_GREP "{"
+$S_ECHO "$PHP_STATUS" | $S_GREP "{"
+exit 0

zabbix/zabbix_php_fpm_template_4.0.xml

@@ -89,7 +89,7 @@
                     <type>0</type>
                     <snmp_community/>
                     <snmp_oid/>
-                    <key>php-fpm.discover</key>
+                    <key>php-fpm.discover[{$PHP_FPM_STATUS_URL}]</key>
                     <delay>2m</delay>
                     <status>0</status>
                     <allowed_hosts/>