EasyList

Archive for September 2011

Announcing EasyList Italy

written by MonztA, on Sep 27, 2011 8:19:00 PM.

Although EasyList author Khrin started to develop EasyList Italy just two weeks ago, it already has a considerable amount of filters and it is now ready to go live. The subscription is designed as a supplement for Italian-language websites and is now listed on the EasyList homepage, from where it may be viewed or installed.

If you want to provide feedback about EasyList Italy, including reporting adverts or problems, please use the Adblock Plus issue reporter or open a topic in the EasyList forums.

Two years, seventeen days, three hours and five minutes

written by Michael, on Sep 10, 2011 12:13:00 PM.

It has been two years, seventeen days, three hours and five minutes since I became involved with EasyList by creating an account on the EasyList forums. When I joined, I did not expect that I would ever rise to my present position of lead EasyList and Privacy author, forum administrator and all the other roles that I have accumulated in my EasyBlog profile.

It was chance that lead to my first major breakthrough: only three months after I had joined the forum, I was granted contributor status and thereby discovered, via a topic in the private forum, that Ares2 was officially struggling to maintain the subscriptions (my 500 posts over my first three months on the forum had nothing to do with this). Wladimir was, at the time, reorganising the Adblock Plus repositories anyway and offered a Mercurial repository to Ares2, which was eventually accepted. The question then arose who would assist Ares2, which I replied to with the following post:

"I would be willing to help, although I appreciate that I am somewhat of a novice in comparison to the other members of the private forum."

Michael

Despite my acknowledgment of my limited involvement with EasyList, my application was accepted, along with Erunno's, and by early December I was actively committing changes.

Becoming the lead author was also a matter of being in the right place at the right time. By the end of April I was effectively the only active EasyList author, to the extent that I complained, via private message to Erunno and Ares2, that:

"I signed up for a group project, not a one man band."

Michael

Although I received assistance from Erunno, Ares2 found himself unable to continue to run EasyList and in June he therefore offered me the position of lead author, which I readily accepted. It was also acknowledged that we need more help, and three people were offered authorship at the time: MonztA, Khrin and Famlam, of whom MonztA and Khrin immediately accepted and soon took on their new positions.

After the peculiar circumstances that lead me to my position, I decided to develop EasyList further, redesigning the website, which had not been significantly altered since Rick was the EasyList author, and adding this blog. I added groups to the forum to more clearly identify significant people, such as contributors, subscription authors and EasyList authors, and added the private "EasyList Authors' Forum" to reduce the excessive use of private messages by authors. I have also been one of two authors of SubscriptionList and helped to develop the Adblock Plus documentation and check the English localisation of Adblock Plus - I am responsible for the description of Adblock Plus on the Mozilla Add-ons domain. More recently, Famlam has taken up my offer to become an author and I have developed FOP (Filter Orderer and Preener), which automatically sorts and commits changes to Adblock Plus subscriptions.

That is not to say that I have no regrets about my time with the project - I have certainly made some significant mistakes, including committing a rule that would have broken a very large number of websites had the filter not been spotted quickly and being overly antagonistic to an important company - but that I am generally proud of my involvement with EasyList.

Perhaps the most significant change to the EasyList project while I have been the lead author is to make the subscriptions available as Tracking Protection Lists (TPLs) for Microsoft's Internet Explorer using a script I developed, but which was later significantly improved by Wladimir; EasyPrivacy TPL now accounts for just over half of all EasyPrivacy downloads.

Those are my memories of EasyList - my reminisces. But I now need to move to the present and to reality. Over my two years with the project I have been successful in real life, but have realised that the resulting changes will reduce my level of my involvement for the foreseeable future to such a low standard as to do a disservice to the community. EasyList needs stability - there has been approximately one lead author a year since Rick's unfortunate death in 2009 - and I am unable to provide it. As I hinted in a hidden message in a recent blog post, I am therefore stepping down from all of my roles associated with EasyList and Adblock Plus.

MonztA has kindly agreed to take over the role of lead EasyList and EasyPrivacy author (he is already the lead EasyList Germany author), effective immediately. He has been writing subscriptions for at least twice as long as I've even been involved with EasyList and I know that he already has plans to continue to develop the project. I wish him and the community the best of luck and thank the numerous people (too many to list here) who have supported me as an author. But now I must leave. After two years, seventeen days, three hours and five minutes.

New EasyList author: Famlam

written by Michael, on Sep 9, 2011 3:02:24 PM.

I am pleased to announce that we have a new author for EasyList, EasyPrivacy and EasyList Germany: Famlam. He has been a member of the EasyList forum for nearly a year and a half and is already a contributor to AdBlock for Chrome, one of several advert blocking add-ons for Chrome that use the EasyList subscriptions. He was officially granted the status of EasyList author yesterday evening and I hope that you will welcome him into his new role.

I would also encourage anyone else interested in becoming an author for EasyList, EasyPrivacy or EasyList Germany to contact the project. As I noted in a recent article on EasyList usage and downloads, we have very few authors relative to the millions of users of the subscriptions and would be glad of additional assistance.

EasyList Statistics: August 2011

written by Michael, on Sep 1, 2011 12:00:00 PM.

Looking back, it has been a long time since anyone has evaluated or released the statistics for the EasyList subscriptions, so I thought that, as August has just eneded, it might be time to estimate exactly how many users there are of the filter lists.

Even though I am aware that most people are interested simply in the user estimates, I should start by explaining how the statistics are gathered for the users of EasyPrivacy and any other privacy conscious individuals. When a browser connects to a web server, it automatically sends information to request a copy of, in this instance, a text file. This request includes the user agent, which for Firefox states the name of the operating system (although this information is not present in my statistics) and the version of the browser. The Adblock Plus website, like so many others, logs these requests with the time and date the request was made. These logs are then analysed by AWStats. No active tracking scripts have been installed.

Although there is a lot of information, calculating the number of users of the EasyList subscriptions cannot provide a complete picture for a variety of reasons. Adblock Plus updates subscriptions between four and six days after they were last updated. However, if a user does not use a browser for a day, then an update will be delayed for a day, up to ten days after a subscription was last updated. This being August, many people are likely to be on holiday, which means that the subscription updates are likely to be less frequent than my assumed value of five days. It is also possible, but extremely unlikely, that results may be overestimated as a result of many people forcing a download from the Adblock Plus preferences window.

Viewing the statistics at the time this algorithm was implemented, Wladimir hoped that the changes would prevent a “Monday peak” from overloading the server, the result of office workers all updating on Monday morning if the five day interval passed over the weekend. However, it is clear that there are a number of users of the subscriptions who do not have a recent version of Adblock Plus installed. Using the four full weeks of August 2011 as an example, Mondays show on average 12% more downloads than would be expected (week total / 7), ranging from 110.8% to 118.5% of the expected traffic. In contrast, downloads tended to tail off as the week progressed, the most marked drop being on Saturdays and Sundays, with an average of 92% of the expected traffic each day. This ranged from a low of 91.6% on Saturday 13 August to a weekend high of 95.8% on Sunday 28 August.

In spite of the variation based on week day the total number of downloads per week was fairly stable: in order, each week achieved 98.1%, 99.0%, 100.7% and 102.2% of expected traffic, the initial low results likely the results of holidays without computers and therefore without updates to EasyList.

Now, however, it is time to answer the big question: how many people actually use the subscriptions? These values were calculated by taking the total number of downloads in a month, dividing the resulting total by thirty one to find the average number per day and then multiplying by five, as this would be, if everyone used their browser daily (which they don't), the average update period.

Going through the subscriptions, it is no surprise that EasyList has the largest user base. Taking into account all combined subscriptions, versions and variations, such as the one with element hiding rules, the one without rules for adult websites and relevant tracking protection lists, we find that there were seventy three million downloads of the file in August, suggesting that there are twelve million users, of whom 99% use Adblock Plus filter format. EasyList Germany has also been popular, with two and a half million users (99.9% Adblock Plus format), as has EasyPrivacy, despite the comparatively paltry total of "just" half a million users. However, unlike EasyList and EasyList Germany, which have seen little usage as Tracking Protection Lists, just over half of the users of EasyPrivacy use EasyPrivacy TPL, its popularity likely due to Microsoft's promotion of the list, which we are very grateful for.

Next, it is possible to put these values in perspective: how dominant is EasyList as an Adblock Plus subscription? Looking at the public statistics page for the add-on and ignoring the recent erroneous readings, we find that there are twelve and a half million users of Adblock Plus. Taking into account the usage of AdBlock for Chrome and Adblock Plus for Chrome, which according to user agents accounts for, at most, 15% of installations, we find that 80% of people using Adblock Plus use EasyList as a primary subscription. This is despite the random selection of EasyList or Fanboy's List for a new English language user. I would attribute this popularity to historical reasons (people installed Adblock Plus when EasyList was the only recommended subscription for English language users); the large number of recommended supplementary subscriptions, including EasyList Germany, RU AdList, Liste FR and ChinaList [*], to name but a few, which are selected automatically when the subscription and browser languages match; and its position as a fallback subscription: if the browser language does not match any recommended subscription, EasyList is automatically installed.

Of course, the minor subscriptions also deserve a mention. EasyList without element hiding has accumulated around seven thousand users, EasyList without rules for adult sites has over two thousand users and EasyPrivacy without international filters has just over one thousand users. Finally, EasyTest, an EasyList supplement used for examining filters known to cause problems on websites, has accumulated just 33 users.

While there is great variation in the usage of the EasyList subscriptions, the statistics emphasise the great responsibility that we authors have - a minor mistake can quickly affect thousands of people. I would therefore appeal to anyone who has made it to the final paragraph to consider getting involved with the project to assist the very few authors we have - even suggesting rules for problematic sites helps us to reduce the time it takes to commit a fix. Regardless of this challenge, I am optimistic: provided the requisite bandwidth is made available (4.2 TB was required this month for the Adblock Plus version of EasyList alone), I see no reason why the project should not continue to be successful.

[*]

I am not going to discuss the usage of the affiliated subscriptions as I do not have the statistics for the files hosted third-party, but the subscription authors are welcome to write their own analysis with the assistance of statistics for the EasyList mirrors - you should know how to access the logs, if not, please private message me.