[closed] Bot detection with error emails

mrfelna · April 6, 2015, 12:23am

While running FarCry 7.0.6, I’ve received several error emails caused by a googlebot visit but having “(not a bot)” in the subject. In fact the only error email received with “(bot)” in the subject was when the user agent was empty. Is there a configuration option I need to set or is this a bona fide bug?

Cheers!

P.S. Sorry for the inactivity recently. Been very busy

justincarter · April 6, 2015, 11:11am

I’m not sure why bot detection would be failing but I can point you in the right direction if you want to debug it.

The bot detection will be checked on each request, here:

github.com

farcrycore/core/blob/p700/Application.cfc#L52


		<cfif len(FARCRY_DSN)>


			<!--- set the farcry dsn, dbtype and dbowner --->
			<cfset THIS.dsn = FARCRY_DSN /> 
			<cfset FARCRY_DBTYPE = system.getEnv("FARCRY_DBTYPE")>
			<cfif NOT isNull(FARCRY_DBTYPE)>
				<cfset THIS.dbType = FARCRY_DBTYPE>
			</cfif>
			<cfset FARCRY_DBOWNER = system.getEnv("FARCRY_DBOWNER")>
			<cfif NOT isNull(FARCRY_DBOWNER)>
				<cfset THIS.dbOwner = FARCRY_DBOWNER>
			</cfif>


			<!--- set up the datasource settings --->
			<cfset this.datasources[FARCRY_DSN] = {
				"class" = system.getEnv("FARCRY_DSN_CLASS"),
				"connectionString" = system.getEnv("FARCRY_DSN_CONNECTIONSTRING"),
				"database" = system.getEnv("FARCRY_DSN_DATABASE"),
				"driver" = system.getEnv("FARCRY_DSN_DRIVER"),
				"host" = system.getEnv("FARCRY_DSN_HOST"),
				"port" = system.getEnv("FARCRY_DSN_PORT"),

If “googlebot” is present in the user agent string then it should be detected as a bot, as it’s in the default list of bot user agents (just above on line 44).

It’s possible to turn off bot detection by placing this.botDetection = false in your farcryConstructor.cfm, but I’m assuming that’s not the case in your app.

It seems that the user agent match is case sensitive, although the user agent string is passed to lcase() when reFindAny() is called so that shouldn’t be it:

github.com

farcrycore/core/blob/p700/Application.cfc#L170


		
		<cfset var z = 0 />
		<cfset var q = 0 />
		<cfset var commandString = "" />
		<cfset aStates = arrayNew(1) />
		<cfset sStates = "" />
		<cfset returnArray = arrayNew(1) />
		
		
		<cfset aStates = listToArray(arguments.stateCommandString, ":") />
		
		<!--- 
			The reason this gets a bit complicated is to be backwards compatable. 
			If they just passed a string like "+one,two,three" we'll assume they want
			the core default agents, and want to add to them. --->
		<cfif not len(stateCommandString) or left(stateCommandString, 1) eq "+">
			<cfset sStates = arguments.asteriskDefaults />
		<cfelse>
			<cfset sStates = "" />
		</cfif>

Please let us know what you find

justincarter · April 6, 2015, 11:17am

Oh and specifically related to the data present in an error, this is where the bot detection is checked and turned into the string “bot” or “not a bot” (also worth noting, we shouldn’t be using IIF() and DE() in core):

github.com

farcrycore/core/blob/p700/packages/lib/error.cfc#L104


		<cfif isdefined("application.sysInfo.machineName")>
			<cfset stResult["machinename"] = application.sysInfo.machineName />
		<cfelse>
			<cfset stResult["machinename"] = "Unknown" />
		</cfif>
		<cfif isdefined("application.sysInfo.machineName")>
			<cfset stResult["instancename"] = application.sysInfo.instanceName />
		<cfelse>
			<cfset stResult["instancename"] = "Unknown" />
		</cfif>
		<cfif isdefined("application.sysInfo.version.string")>
			<cfset stResult["coreversion"] = application.sysInfo.version.string />
		<cfelse>
			<cfset stResult["coreversion"] = "Unknown" />
		</cfif>
		<cfset stResult["bot"] = IIF(!isdefined("request.fc.hasSessionScope") || !request.fc.hasSessionScope,DE("bot"),DE("not a bot")) />
		<cfset stResult["browser"] = cgi.HTTP_USER_AGENT />
		<cfset stResult["datetime"] = now() />
		<cfset stResult["host"] = cgi.http_host />
		<cfset stResult["httpreferer"] = cgi.http_referer />
		<cfset stResult["scriptname"] = cgi.script_name />

mrfelna · April 7, 2015, 12:04am

Thanks for the pointers. I’ll do some digging, see if I can find anything and share anything worth sharing.

I couldn’t find this.botDetection in the farcryConstructor.cfm, but I did find <cfparam name="this.botAgents" default="*:+bingbot" /> in Application.cfm. If I understand __plusMinusStateMachine correctly, this should add bingbot to the default list.

Here’s the googlebot user agent in question: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Initial investigation indicates that the bot agents array is empty for some reason …

justincarter · April 7, 2015, 12:20am

Yes I also read "*:+bingbot" as adding bingbot to the array. Interesting to note that bingbot is missing from core though, we should definitely add it

Is that an Application.cfm in your project folder? I’d be surprised if that was still having any effect, FarCry Core has used Application.cfc for quite a few major versions now. If it’s specified in your projects Application.cfc then perhaps that is the problem, the line of code might need to be moved into the farcryConstructor.cfm instead?

mrfelna · April 7, 2015, 12:23am

It is in Application.cfc which extends="farcry.core.Application". Does this not work?

I’ve found the issue - minor bug in core Application. Just tring to test before I create a pull request.

mrfelna · April 7, 2015, 12:42am

Bug fixed - I hope!

justincarter · April 7, 2015, 1:02am

I think the preferred location to configure it is in your farcryConstructor.cfm:

<cfset this.botAgents = "*:+bingbot">

Your pull request is spot on though, seems like it’s been this way since mid-2010:

Thanks for the fix It’s merged, and I’ll backport it to the p700 branch as well.

mrfelna · April 7, 2015, 1:16am

Thanks! Got a couple of questions about this - where’s the best place to put them?

Obviously just me that cares then

My pleasure. I’m just sorry I have been and am still too busy to do much else recently.

As always, I am open to constructive criticism

justincarter · April 7, 2015, 1:24am

Anywhere in your projects farcryConstructor.cfm should be fine, e.g. just after the data source and plugin configuration…

<cfset this.botAgents = "*:+bingbot">

Also, I’ve just added bingbot to the default bot list for p700 and p710:

https://github.com/farcrycore/core/commit/9592f2a

mrfelna · April 7, 2015, 1:29am

My mistake - I meant where is the best place to put the questions, not that line of code.

Nice! That list may need an overhaul but not now. If I get time, I may have a bash.

justincarter · April 7, 2015, 1:55am

Ahh, questions about the farcryConstructor.cfm? Perhaps in a new thread here

A review of current / commonly used bot user agents would be a good idea.

This also needed another patch, now that this.botIPs is an array and not a string: