testdb=#  \d test_table_2
                                        Table "public.test_table_2"
   Column    |          Type          | Collation | Nullable |                   Default
-------------+------------------------+-----------+----------+---------------------------------------------
 id          | integer                |           | not null | nextval('your_table_name_id_seq'::regclass)
 domain_name | character varying(255) |           |          |
Indexes:
    "your_table_name_pkey" PRIMARY KEY, btree (id)
    "test_index" btree (domain_name)
testdb=#
testdb=#
testdb=#  explain analyze select * from test_table_2 where domain_name like 'abcd%';
                                                         QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------
 Gather  (cost=1000.00..13576.05 rows=100 width=25) (actual time=78.204..80.452 rows=0 loops=1)
   Workers Planned: 2
   Workers Launched: 2
   ->  Parallel Seq Scan on test_table_2  (cost=0.00..12566.05 rows=42 width=25) (actual time=50.570..50.571 rows=0 loops=3)
         Filter: ((domain_name)::text ~~ 'abcd%'::text)
         Rows Removed by Filter: 333443
 Planning Time: 2.578 ms
 Execution Time: 80.777 ms
(8 rows)

This is because most databases are not initialized with locale C and are unable to utilize the default operator class for pattern matching.

If you do not already know, you need to be clear about 2 terms - Locale and Operator class.

Locale:
Locale refers to the specific regional or cultural settings used by a computer system, including language, date and time formats, and collation rules. Each locale has its own set of rules for sorting and comparing characters.
You can check the locale that the database uses via the below.

testdb=# SHOW LC_COLLATE;
 lc_collate
-------------
 en_US.UTF-8
(1 row)

Operator class:
An operator class in PostgreSQL is a way to define a custom sorting and comparison behaviour for a specific data type, such as text, integer, or double. Operator classes are used to create custom functions that can be used as operators in SQL queries, allowing you to extend the functionality of the default operators.

If you wish to read more, here is a very beautiful explanation on operator classes. So, basically if your DB is initialized with locale C, you can leverage the default operator classes for ordinary comparisons as well as pattern matching expressions. But, if your DB is initialized with some other locale, you can leverage the default operator classes for ordinary comparisons but not for pattern matching expressions. For pattern matching expressions, you can use some inbuilt operator classes beside the default operator class. Here is an excerpt from Postgres documentation:

*The operator classes text_pattern_ops, varchar_pattern_ops, and bpchar_pattern_ops support B-tree indexes on the types text, varchar, and char respectively. The difference from the default operator classes is that the values are compared strictly character by character rather than according to the locale-specific collation rules. This makes these operator classes suitable for use by queries involving pattern matching expressions (LIKE or POSIX regular expressions) when the database does not use the standard “C” locale. As an example, you might index a varchar column like this:
CREATE INDEX test_index ON test_table (col varchar_pattern_ops);

Note that you should also create an index with the default operator class if you want queries involving ordinary <, <=, >, or >= comparisons to use an index. Such queries cannot use the xxx_pattern_ops operator classes. (Ordinary equality comparisons can use these operator classes, however.) It is possible to create multiple indexes on the same column with different operator classes. If you do use the C locale, you do not need the xxx_pattern_ops operator classes, because an index with the default operator class is usable for pattern-matching queries in the C locale.*

So, now when you create an index with the specific operator class like below, it is evident that index scan kicks in.

testdb=# CREATE INDEX test_index ON test_table_2 (domain_name varchar_pattern_ops);
CREATE INDEX
testdb=#
testdb=# \d test_table_2
                                        Table "public.test_table_2"
   Column    |          Type          | Collation | Nullable |                   Default
-------------+------------------------+-----------+----------+---------------------------------------------
 id          | integer                |           | not null | nextval('your_table_name_id_seq'::regclass)
 domain_name | character varying(255) |           |          |
Indexes:
    "your_table_name_pkey" PRIMARY KEY, btree (id)
    "test_index" btree (domain_name varchar_pattern_ops)

testdb=#
testdb=#  explain analyze select * from test_table_2 where domain_name like 'abcd%';
                                                         QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------
 Index Scan using test_index on test_table_2  (cost=0.42..8.45 rows=100 width=25) (actual time=0.488..0.489 rows=0 loops=1)
   Index Cond: (((domain_name)::text ~>=~ 'abcd'::text) AND ((domain_name)::text ~<~ 'abce'::text))
   Filter: ((domain_name)::text ~~ 'abcd%'::text)
 Planning Time: 86.148 ms
 Execution Time: 0.517 ms
(5 rows)

So here, in the context of text_pattern_ops, the operator class is used to define custom comparison and sorting behavior for the text data type when performing pattern-matching queries using the LIKE operator or POSIX regular expressions.

So far so good. When the query is of the form LIKE 'abcd%', there is no problem since index utilization happens without issues as evident above.

But, what if the query is of the form LIKE '%abcd' for the same index as above?

testdb=#  explain analyze select * from test_table_2 where domain_name like '%abcd';
                                                         QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------
 Gather  (cost=1000.00..13576.05 rows=100 width=25) (actual time=69.966..71.298 rows=0 loops=1)
   Workers Planned: 2
   Workers Launched: 2
   ->  Parallel Seq Scan on test_table_2  (cost=0.00..12566.05 rows=42 width=25) (actual time=46.064..46.064 rows=0 loops=3)
         Filter: ((domain_name)::text ~~ '%abcd'::text)
         Rows Removed by Filter: 333443
 Planning Time: 0.207 ms
 Execution Time: 71.753 ms
(8 rows)

Clearly, index scan is not done and we see a sequential scan happening.

Why is this?

This is because

B-tree indexes are well-suited for handling ordered data and range queries, but they have limitations when it comes to wildcard matching at the beginning of the search pattern. In B-tree indexes, the index entries are sorted in a specific order, allowing efficient range queries by traversing the tree structure. However, wildcard matching at the beginning of the pattern requires scanning the entire index because the index entries are ordered based on their values from left to right.

When you use a LIKE query with a wildcard character at the end, such as "LIKE 'abcd%'", the B-tree index can efficiently determine the range of values that satisfy the query condition, starting from the specified prefix. It can traverse the index entries in the sorted order and efficiently identify the matching values.

On the other hand, when you use a LIKE query with a wildcard character at the beginning, such as "LIKE '%abcd'", the B-tree index cannot effectively leverage its sorting order to narrow down the search space. As a result, it needs to scan the entire index to find the matching values, which can be less efficient compared to a sequential scan of the table itself.

To put it simply, if the wildcard character is at the end, I know the starting characters so I can traverse the tree from the root till the point where the wildcard character appears so I can use the index.

But, if the wildcard character is at the beginning, how do I traverse the tree since I do not know which characters might come in the beginning and how many characters might come, so a tree traversal is not possible and hence index is not used.

To deal with such cases, we can use another type of indexes called trigram indexes.

Trigram indexes work by breaking up text in trigrams or 3-letter sequences and the actual indexes themselves must be a GIN or GIST index.

GIST helps with more cases than GIN, takes less index building time but more querying time.

GIN takes more index building time but less querying time.

So, which one to use is completely dependent on your use case. For most simple cases, GIN outperforms GIST by a large margin.

Also, before you use them, you will need to enable pg_trgm extension in your DB.

Here is a very good explanation on Trigram Indexes. Postgres' documentation is pretty lucid as well.

testdb=# create index test_index on test_table_2 using gin (domain_name gin_trgm_ops);
CREATE INDEX
testdb=#
testdb=#
testdb=# \d test_table_2
                                        Table "public.test_table_2"
   Column    |          Type          | Collation | Nullable |                   Default
-------------+------------------------+-----------+----------+---------------------------------------------
 id          | integer                |           | not null | nextval('your_table_name_id_seq'::regclass)
 domain_name | character varying(255) |           |          |
Indexes:
    "your_table_name_pkey" PRIMARY KEY, btree (id)
    "test_index" gin (domain_name gin_trgm_ops)

testdb=#
testdb=#
testdb=# explain analyze select * from test_table_2 where domain_name like 'abcd%';
                                                      QUERY PLAN
----------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on test_table_2  (cost=68.78..435.05 rows=100 width=25) (actual time=2.136..2.139 rows=0 loops=1)
   Recheck Cond: ((domain_name)::text ~~ 'abcd%'::text)
   Rows Removed by Index Recheck: 4
   Heap Blocks: exact=4
   ->  Bitmap Index Scan on test_index  (cost=0.00..68.75 rows=100 width=0) (actual time=1.679..1.681 rows=4 loops=1)
         Index Cond: ((domain_name)::text ~~ 'abcd%'::text)
 Planning Time: 3.563 ms
 Execution Time: 2.262 ms
(8 rows)

testdb=#
testdb=#
testdb=# explain analyze select * from test_table_2 where domain_name like '%abcd';
                                                      QUERY PLAN
----------------------------------------------------------------------------------------------------------------------
 Bitmap Heap Scan on test_table_2  (cost=52.78..419.05 rows=100 width=25) (actual time=1.013..1.014 rows=0 loops=1)
   Recheck Cond: ((domain_name)::text ~~ '%abcd'::text)
   Rows Removed by Index Recheck: 2
   Heap Blocks: exact=2
   ->  Bitmap Index Scan on test_index  (cost=0.00..52.75 rows=100 width=0) (actual time=0.839..0.840 rows=2 loops=1)
         Index Cond: ((domain_name)::text ~~ '%abcd'::text)
 Planning Time: 0.171 ms
 Execution Time: 1.056 ms
(8 rows)

After adding trigram index, it is clear that index is utilized for scanning in both the cases - wildcard at the beginning and wildcard towards the end. In comparison with a normal B-Tree index with xxx_pattern_ops operator class, for LIKE 'abcd%' type queries, the former does perform better but then the former has limited use cases.Trigram indexes give you an index advantage in a wide variety of cases on the other hand and come in handy in some really tricky situations. So, choose wisely and see if you can leverage both for your use case.

Happy indexing!

• Want users to always visit your site on HTTPS instead of HTTP.

• Have a shortened URL and want users to visit the actual site when the short URL is used.

• Have your content on one website but have an alias for the same.

• Have a new home for your website and want users to no longer visit the old one.

There are multiple ways to achieve it where each method serves its own purpose and should be leveraged according to the specific needs. Some of the popular ways are listed below.

301/302/307/308 redirection

This URL redirection is unmasked redirection. It is called so because the target URL is not masked and visible to the user and the user is made aware of it.

It needs that the response HTTP status code is 301/2/7/8 and a Location header is passed in the response headers.

Here's an excerpt from the RFC itself:

The server SHOULD generate a Location header field in the response containing a preferred URI reference for the new permanent URI. The user agent MAY use the Location field value for automatic redirection.

The browser is smart for a lot of reasons and this is one of it. It sees that the status code is 3XX, understands that redirection needs to kick in and then looks for the Location header in the response headers. It then visits the target URL mentioned by the Location header through the usual process of visiting a URL.

301 and 308 are used for permanent redirections whereas 302 and 307 are used for temporary redirections.

In permanent redirections, the target website is indexed by the search engines whereas in temporary redirections, the search engines are made aware that this redirection is on a temporary basis and indexing needs to be set for the source URL itself.

You can read more on the specific use cases of each here.

The difference between 301-308 or 302-307 is that the first set of them - 301/2 do not explicitly ask the user agent to change the METHOD during redirection whereas the latter ones - 307/8 explicitly ask the user agent to not change the METHOD during redirection. It is just a contract enforcement mechanism.

You can read more on this here.

Frame redirection

This URL redirection is masked redirection. It is called so because the target URL is masked and the user is not made aware of it explicitly.

It works by embedding the entire content of a target site into a frame and generating a new page. This page will be served by a domain forwarding service which ensures that this works seamlessly.

When an internet user visits the source URL, the resultant IP as part of the DNS resolution is that of the domain forwarding service. The service sees that someone needs to visit the source URL, checks the mapped target URL for the same on its end, takes the content of the target URL, embeds it into a frame and then generates and returns the page.

One important thing to note in this kind of redirection is the non-changing URL in the address bar.

If abc.com is fetching content for xyz.com, and if the user clicks on some link on the page, the URL in the address bar would not change to abc.com/some-path even when the content will be fetched from xyz.com/some-path.

The reason the address bar does not reflect the path when using frames is because of how web browsers handle frames. The URL in the address bar is associated with the main document and not the individual frames within that document. Each frame is a separate HTML document with its own URL, but this URL is not displayed in the address bar of the browser.

When a frame is loaded, it's essentially a separate webpage being loaded within the main webpage. The browser treats each frame (and the content within) as a separate document. This means that when you navigate within a frame, the URL of the main document (which is what's displayed in the browser's address bar) doesn't change.

This redirection is not to be confused with CNAME redirection which is also a masked redirection.

CNAME redirection

This kind of redirection is masked.

The DNS lookup for a query when a URL is entered in the browser doesn't complete till an A/AAAA record is found since these are the ones that provide an IP for the browser to connect with.

A CNAME record involves a source and a destination and implies that every time an authoritative IP is fetched for the source domain, the target domain's resolution needs to kick in and its IP needs to be returned for that of the source domain.

So, this redirection works at the DNS level itself and hence, needless to say, the web server which sits behind that IP will need to handle requests for both domains. This is different from frame redirection where the web server doesn't have to handle requests from the source domain.

You can read more on this and the above 2 methods as well here.

Refresh Meta tag and HTTP refresh header

This kind of redirection is unmasked.

Upon adding a refresh meta tag to the HTML document, the browser automatically refreshes the page to the mentioned URL and if you mention the delay as 0, the refresh happens instantaneously simulating a redirection.

This works just like sending the Refresh header in the HTTP response.

So, the web server for the source domain can either serve an HTML page with the refresh meta tag mentioning the target URL or the web server can respond with the refresh header in the response and then the browser can serve the target URL page.

And if you are trying to do it via Cloudformation like below:

WebACL:
  Type: "AWS::WAFRegional::WebACL"
  Properties:
    DefaultAction:
      Type: BLOCK
    MetricName: "MyWebACL"
    Name: MyWebACL
    Rules:
      - Action:
          Type: ALLOW
        Priority: 1
        RuleId: !Ref RateBasedRule


RateBasedRule:
  Type: "AWS::WAFRegional::RateBasedRule"
  Properties:
    Name: MyRateBasedRule
    MetricName: "MyRateBasedRule"
    RateKey: "IP"
    RateLimit: 2000
    MatchPredicates:
      - DataId: !Ref IPSet
        Negated: false
        Type: "IPMatch"

This is not going to work since rate-based rule creation is supported via Cloudformation but association is not. This is a known issue at AWS end.

You can either do the association via console or else use AWS cli for the same if it is part of an automation.

You will need to first fetch a change token which gets associated with the change you are making to the WebACL.

When you want to create, update, or delete AWS WAF objects, get a change token and include the change token in the create, update, or delete request. Change tokens ensure that your application doesn't submit conflicting requests to AWS WAF.

Each create, update, or delete request must use a unique change token. If your application submits a GetChangeToken request and then submits a second GetChangeToken request before submitting a create, update, or delete request, the second GetChangeToken request returns the same value as the first GetChangeToken request.

When you use a change token in a create, update, or delete request, the status of the change token changes to PENDING , which indicates that AWS WAF is propagating the change to all AWS WAF servers. Use GetChangeTokenStatus to determine the status of your change token.

$ change_token=$(aws waf-regional get-change-token --output text) # this line is needed so that the output stored in the variable isn't enclosed in quotes
$ aws waf-regional update-web-acl --web-acl-id ${web_acl_id} --change-token ${change_token} --updates Action="INSERT",ActivatedRule="{Priority=1,RuleId=${rule_id},Action={Type=\"BLOCK\"},Type=\"RATE_BASED\"}"

References

https://docs.aws.amazon.com/cli/latest/reference/waf-regional/get-change-token.html

https://docs.aws.amazon.com/cli/latest/reference/waf-regional/update-web-acl.html

CGI or Common Gateway Interface is a standard protocol that defines how web servers and external programs can communicate. It allows web servers to execute programs written in various languages, such as Perl, Python, and PHP, in response to web requests. CGI scripts are typically used to generate dynamic web pages, process form data, and access databases.

Working

Here's how CGI works:

• A web browser sends a request to a web server for a specific URL.

• The web server checks if the URL maps to a static file (e.g., HTML, CSS, image).

• If it's not a static file, the web server searches for a CGI script that matches the URL.

• If a CGI script is found, the web server launches the script and passes it any parameters that were included in the request URL.

• The CGI script executes and generates output (e.g., HTML code).

• The web server sends the output back to the web browser.

Despite its advantages, CGI has some limitations. One of the main issues is performance. Every time a CGI script is executed, a new process is created on the server. This can consume significant resources, especially if there are many simultaneous requests. This led to the development of alternative solutions, one of which is mod_perl.

Mod_perl

mod_perl is an Apache module (service programs that can be dynamically linked and loaded to extend the nature of the HTTP Server) that embeds the Perl interpreter directly into the web server. This allows Perl scripts to be executed much faster than CGI scripts, as they don't need to be launched as separate processes.

Working

With mod_perl, the Perl interpreter is started only once when the server starts. Perl scripts are loaded into memory, and subsequent requests are handled by the persistent interpreter. This eliminates the overhead of starting a new Perl process for each request, resulting in faster execution and reduced resource consumption.

mod_perl also extends the Apache API, allowing developers to write Apache modules entirely in Perl. This gives developers access to all stages of the request processing cycle and allows them to manipulate Apache's internal tables and state mechanisms. This level of control and integration is not possible with traditional CGI.

To ease the transition from CGI to mod_perl, it includes features to run existing CGI scripts under mod_perl with little or no modification. For example, Apache::Registry and Apache::PerlRun are two mod_perl modules that can execute CGI scripts much faster than traditional CGI, because they take advantage of the persistent Perl interpreter embedded in the server.

When Apache receives a request, it processes it in 12 phases. The advantage of breaking up the request process into phases is that Apache gives a programmer the opportunity to hook into the process at any of those phases. For every phase a standard default handler is supplied by Apache.

Modules take control of request processing at each of the phases through a set of well-defined hooks provided by Apache. The subroutine or function in charge of a particular request phase is called a handler. Apache also provides modules with a comprehensive set of functions they can call to achieve common tasks including file I/O, sending HTTP headers or parsing URIs. These functions are collectively knows as Apache API.

Like other Apache modules, mod_perl is written in C, registers handlers for request phases and uses the Apache API. However, mod_perl doesn't directly process requests. Rather, it allows you to write handlers in Perl. When the Apache core yields control to mod_perl through one of its registered handlers, mod_perl dispatches processing to one of the registered Perl handlers.

The <Location> section in the Apache configuration (httpd.conf) assigns a number of rules that the server follows when the request's URI matches the location.

<Location /foo> SetHandler modperl PerlResponseHandler FooServer </Location>

This configuration causes all requests for URIs starting with /foo to be handled by the mod_perl Apache modules with the handler from the FooServer perl module.

Directives :-

SetHandler

SetHandler set to perl-script or modperl tells Apache that mod_perl is going to handle the response generation.

PerlResponseHandler

This tells mod_perl to use the FooServer perl module to handle the response generation.

By default, the mod_perl API expects a subroutine named handler() to handle the request in the registered Perl*Handler module. Thus, if your module implements this subroutine, you can register the handler with mod_perl by just specifying the module name.

DNS Zone Files

A DNS Zone file is a plain text file stored on a controlling DNS server that contains all the records for every domain within a given zone. Zone files can include many different record types, but must always begin with what is called an SOA record (Start of Authority).

Types of Records

As mentioned, there are a handful of different types of records used within a DNS Zone, all of which serve a unique purpose. Below are some examples of the most commonly used record types and a brief description of each.

Start of Authority (SOA)

The first record in any zone file is the SOA resource record. This record is an essential part of the DNS zone file. It indicates the domain’s zone and the fundamental properties of the domain name server. It is the mandatory record that must be there in all zone files. It specifies the main properties and characteristics of a domain. Each zone file can contain only one SOA record.

Name Server (NS)

These records specify the DNS server responsible for this domain. These records tell recursive name servers which name servers are authoritative for a zone. Recursive name servers look at the authoritative NS records to facilitate which server to ask next when resolving a name.

Mail Exchange (MX)

MX records, usually two, are responsible for specifying which mail server is in charge of receiving email messages on behalf of a site. The email client tries to make an SMTP connection to the primary mail server listed in the zone file. The records are ranked by priority from lowest to highest with the lowest being the primary. If the primary server is not available, the next listed mail server will attempt a routing connection. MX records must point to a domain, not an IP.

Address (A)

The A record or address record is used to find the IP associated with a domain name. This record routes info from the server to the end client’s web browser. It is used to add IP address for a hostname in a zone. It is the highly used resource record in a zone file. Also when you dig for a domain, the default answer you get is a A record which is denoted by a capital A.

AAAA

The quadruple A record has the same function as the A record but is used specifically for the IPv6 protocol.

Canonical Name (CNAME)

This record will alias one site name to another. The DNS lookup will then route domain name requests the new name that the A record holds. These records must point to a fully qualified domain name.

NAME RR VALUE

--------------------------------------------------

xyz.yourdomain.com CNAME abc.yourdomain.com

abc.yourdomain.com A 172.16.142.34

In the above shown example CNAME entry, if you want to reach "xyz.yourdomain.com", your computer's DNS resolver will first fire an address lookup for "xyz.yourdomain.com", and on finding the CNAME record of "abc.yourdomainc.com", your resolver will again fire an address lookup for "abc.yourdomain.com".

Alias Record (ALIAS)

The ALIAS record is functionally similar to a CNAME record in that it is used to point one name to another. An ALIAS record is used to lead the apex domain name (example.com) to a subdomain such as host.example.com. The authoritative nameservers for the Apex domain will subsequently resolve the IP of the hostname to direct traffic.

Text (TXT)

TXT records hold the free-form text of any type. Initially, these were for human-readable information about the server such as location or data center. Presently, the most common uses for TXT records today are SPF and Domain_Keys(DKIM).

Service Locator (SRV)

Generalized service location record, used for newer protocols instead of creating protocol-specific records such as MX. This type of record, while helpful, is not commonly used.

Pointer (PTR)

Pointer records point an IP to a canonical name and used explicitly in reverse DNS. It is important to note that a reverse DNS record needs to be set up on the authoritative nameservers for the person that owns the IP, not the person that owns the canonical name.

Well, not really but that's the general complaint on the release of the news.

Here is a very good explanation of how .ZIP TLD can be a security threat by Bobby Rauch.

And here is the counter argument for it by Eric Lawrence.

My thoughts

I personally feel this is a valid threat because of the simple fact that most users are not technically aware to the extent of even being able to understand how this is a threat.

Next, examples like .SH are not really valid for these TLDs because again, common users are far more likely to download a .ZIP file on a casual basis than a .SH file.

That said, the HSTS argument is completely valid (and is a big silver lining provided that those unicode characters won't be disallowed) and will get the malicious domains suspended or marked as spam but these domains can still do some damage in those short windows.

Why are these TLDs even made available?

Well, they are made available definitely not with the intent to make internet more unsafe. But TLDs like these help the internet in multiple ways.

ChatGPT does a much better job in explaining this, so here's its answer:

The introduction of new generic top-level domains (gTLDs) serves several purposes:

1. Increased Domain Name Availability: The primary motivation behind releasing new gTLDs is to expand the pool of available domain names. With the rapid growth of the internet and the exhaustion of traditional gTLDs like .com, .org, and .net, the introduction of new gTLDs allows for more choices and availability of domain names.

2. Branding and Differentiation: New gTLDs enable businesses, organizations, and individuals to create domain names that are more specific and relevant to their brand or industry. For example, a company in the automotive industry might choose a .auto domain, while a non-profit organization might opt for a .ngo domain. This allows for better branding, differentiation, and recognition within their respective sectors.

3. Market Competition and Innovation: The introduction of new gTLDs promotes competition among domain registries, fostering innovation and new business opportunities. It encourages registry operators to offer unique services and features tailored to specific industries or communities. This can lead to enhanced user experiences, specialized online communities, and improved online services.

4. Localization and Regional Identity: Some new gTLDs are designed to represent specific geographic regions or communities. For instance, city-based gTLDs like .nyc, .london, and .berlin can be used by businesses and individuals associated with those cities, helping to establish a sense of local identity and online presence.

5. Niche and Industry-Specific TLDs: New gTLDs also cater to specific industries, interests, or communities. For example, there are gTLDs like .photography, .tech, .fashion, and .guru that cater to photographers, technology enthusiasts, fashion brands, and consultants, respectively. These industry-specific domains allow businesses and individuals to align their web presence with their specialized niche or expertise.

Overall, the introduction of new gTLDs aims to provide more options for domain name registration, promote innovation, enhance branding, and create a more diverse and inclusive online environment.

So, that way the .zip top-level domain (TLD) is the perfect fit for organizations specializing in file sharing, storage, and download technology, or for anyone offering speedy and efficient online service. (reference)

What should you do?

Nothing much really except be wary of malicious links more and click links provided by trusted sources and be more stringent in clicking links that contain these TLDs anywhere in their URLs. Easier said than done ha!

Great! Let’s talk about the next usecase.

Problem

Given an array of integers, find the maximum xor subarray. Or simply,
Given a₁, a₂, ....., a_n, find i and j , i <= j, such that a_i xor a_i+1 xor ..... a_j-1 xor a_j is the maximum possible value.

Simple solution

Run two loops and for every combination of i and j, run one more loop to calculate xor from index i to j.

maxSoFar = -1
For i = 1 to N
For j = i to N
xorVal = 1
For k = i to j
xorVal = xorVal ^ a[k]
maxSoFar = max(maxSoFar, xorVal)
print maxSoFar

That is an O(N^{​​​​​​​3})! Can we simplify it a bit?

Let’s start slow.

Better solution

Let’s first see, where we can optimise it.

Now, running two loops to get each combination of i and j is fine. What about running the loop from index i to index j to calculate xor?

Suppose, you have calculated xor from index 2 to 5, and later on, to calculate from index 2 to 7, why would you calculate xor from index 2 to 5 again?

Or suppose, if you want xor from index 1 to 4, you already know, xor from index 2 to 4 in the process of calculating from index 2 to 5 and just need to xor it with 1, right?

So, we need some means of storing things.

Let’s optimise here first.

What we will be using here is called a precomputed array.

Precomputed array is an array which has computations which we make in the starting and use them later on.

What do we need to precompute here?
Let’s see.

What if I store the xor values of index 1 to index i in array[i] and form the array for i = 1 to N.

What I mean is, suppose we have an array,

2, 4, 3, 6, 8, 7

If I can store an array as

{^ indicates xor}

2, 2^4, 2^4^3, 2^4^3^6, 2^4^3^6^8, 2^4^3^6^8^7

pre[i] = pre[i-1] xor a[i]

This is the precomputation we will do, this array will be the precomputed array. But why are we doing this exactly?

Let’s see the properties of xor once again.

Triangular property of xor is,

[A xor B = C] => [B xor C = A] => [A xor C = B]

So, suppose, I need to find, xor from 3rd element to 5th element,

That is,

3 ^ 6 ^ 8.

Now,

2^4^3^6^8 = (2^4) ^ (3^6^8)

Now, with the triangular property, I can write it as

3^6^8 = (2^4^3^6^8) ^ (2^4)

Which is the xor of 2nd and 5th element in the precomputed array.

So, xor from 3rd to 5th element is the xor of pre[2] and pre[5].

So generally speaking,

A_i xor A_i+1 xor .... A_j = pre[i-1] xor pre[j]

Or in speaking terms, the xor of numbers from index i to j, is the xor of (xor of numbers from 1 to i) and (xor of numbers from 1 to j).

Great, so once we build the precomputed array, we can calculate xor from index i to index j, in constant time.

Given this, we need to run just two loops now, and for each pair of i and j, we need to find the max out of all [(xor 1 to i) xor (xor 1 to j)].

So, this, now takes, O(N²).

Still, not enough, we can optimise it more.

Optimised solution

Let’s use tries like in the previous use case.

Let us rewrite the original problem.

Given a₁, a₂, .... a_n, find i and j , i <= j, such that a_i xor a_i+1 xor .... a_j-1 xor a_j is the maximum possible value.

Can now be written (thanks to precomputed array) as,

Given a₁, a₂, .... a_n, find i and j , i <= j, such that (xor from 1 to i) xor (xor from 1 to j) is maximum.

So, basically, since we have values in precomputed array as 1 to i, we need to find two numbers from the precomputed array, whose xor is maximum.

Hold on, wasn’t that the same question in the previous usecase?

Except that, in the previous usecase, we found out two numbers with maximum xor from given array, and here we are trying to find two numbers with maximum xor from precomputed array, since, our precomputed array consists of numbers from 1 to that index of the original array.

That’s it, you find the precomputed array, consider this as the original array and proceed exactly as in the previous usecase.

This is the optimised solution.

The complexity is :

O(NlogMAX) + O(N) [for creating precomputed array] = O(NlogMAX), same as the previous usecase.

We will explore the next usecase in another problem.

Trie can store information about keys/numbers/strings compactly in a tree.

Tries consists of nodes, where each node stores a character/bit. We can insert new strings/numbers accordingly.

Storing numbers in trie

We can store numbers in trie using binary representation.
If we have 4 numbers - 1, 4, 5, 7.

We first write binary representations of the same, which are:

001
100
101
111

And then, for every node, 0 goes on the left hand side and 1 goes on the right hand side.

Lets insert 1, 0001

1. We start of with epsilon (E), which is root.

2. Then first bit is 0. So, go to left of root, create a node.

3. Next bit is 0, go to left of this node and create a node.

4. Next bit is 0, go to left of this node and create a node.

5. Next bit is 1, go to right of this node and create a node.

6. Similarly for other numbers.

While traversing the trie, if we already have a node while inserting 0 or 1, simply go to that node and move on to the next bit.

The number of leaves of the tree are the number of integers we are storing in the trie.

XOR

Exclusive or is a logical operation that outputs true only when inputs differ.

Problem

Given an array, find two numbers whose XOR is maximum in the array.

Simple solution

Run two loops and for every pair combination, calculate the xor. If the xor is greater than max_xor_so_far, it replaces the value in the max_xor_so_far.

Algorithm

A[n] = [a1, a2, a3 …. an]
Max_so_far = -1
For i=1 to n
For j=1 to i
If a[i] xor a[j] > max_so_far
Max_so_far = a[i] xor a[j]
Print max_so_far

But, as we see, this has 2 loops and so takes O(N²).

Optimised solution

Now, we use the properties of xor, bit manipulation and a data structure, trie to optimise this solution.

Let us try exploring the properties of xor.

Now, basic properties:

0 xor 0 = 0
0 xor 1 = 1
1 xor 0 = 1
1 xor 1 = 0

Now, forget about getting maximum xor pair from the array.

What is maximum N-bit number we can get when we xor an N-bit number with another N-bit number?

The answer is simple, it is all 1s.

Or

XXXXXXX xor YYYYYYY = 1111111

So, basically if I have a number say 19, I can represent it in binary form as 10011.

If I wish to xor it with some other and get 11111 (all 1s), what do I xor it with? 🤔

Here is where we use the basic properties.

Let’s call that number X.

So,

19 xor X = 31

Or

10011 xor X = 11111

From left hand side to right hand side,

1st bit of X should be 0, since, 1 xor 0 = 1
2nd bit of X should be 1, since 0 xor 1 = 1

Likewise...

The number should be 01100

So, basically every bit of X should be opposite of every bit of 19 to get 31.

10011
01100
--------
11111

At this stage, we can have an intuition for our original problem. So, basically, for each number in the array, I need to look for some other number in the array whose bits are opposite of this number. Pretty simple right?

But, how do we ensure we have a number with opposite bits "at all places" for each number in the array?

Here’s the interesting part, we, sort of, compromise at places where we are unable to get the opposite bit, so, we take the same bit and move on to the next place.

What we mean by this will be much clearer through the below example.

Suppose, I have an array: 9, 3, 10, 1, 13.

We have already learnt how to insert numbers in a trie, so using that, insert 9 into the trie.

Now, consider number 3.

3 = 0011

So, we need 1100 to make 1111 (maximum possible), that is 0011 xor 1100 = 1111

We have 1, go to it.

Next, we need 1 but don’t have it, so we compromise and take 0 and go to it.

Next, we need 0, so go to it.

Next, we need 0, but don’t have it so we compromise again and take 1 and go to it.

Finally, we get 1001, which is 9.

So, before 3, 9 is the number that will fetch the maximum xor for 3, that is, 3 xor 9 = 10.

This is max_so_far = 10.

Now, insert 3 into the trie.

Now, consider number 10.

10 = 1010

So, we need 0101 to make 1111.

We have 0, so go to it.
We don't have 1, so take 0 and go to it.
We don't have 0, so take 1 and go to it.
We have 1, so take it.

So, finally we have 0011 which is 3 as the number in the array before 10, which gives us the maximum possible xor for 10 (considering numbers appearing before 10 in the array), that is, 9.

As max_so_far = 10 > 9, our max_so_far stays.

Insert 10 into the trie.

Now, consider 1.

1 = 0001

So, we need 1110 to make it 1111.

We have 1, so take it.
We don't have 1 so, take 0.
We have 1 so take it.
We have 0 so take it.

So, we get 1010 which is 10, so for 1, 10 gives us maximum possible xor (considering numbers appearing before 1 in the array), that is, 11.

Max_so_far = 10 < 11, so max_so_far = 11

Insert 1 into the trie.

Now, consider 13.
13 = 1101
So, we need 0010 to make 1111

We have 0 so take it
We have 0 so take it
We have 1 so take it
We don't have 0, so take 1

So, we get 0011, which is 3, so for 13, 3 gives us maximum possible xor, that is, 14.

Max_so_far = 14 > 11, so max_so_far = 14

Insert 13 into the trie.

Now, this is the complete trie with all the numbers.

So, finally the maximum xor we got from the array for a pair of numbers is 14.

Now, let’s see it’s time complexity.

Well, we are inserting N numbers into trie.
Each insertion takes log(MAX) operations.

What is MAX now?

MAX is the number which is maximum possible for given number of bits or say (all 1s).

How to choose it, well, choose the maximum number from the array, 13 in the above case, choose the next power of 2 minus 1, which is 16-1, 15, so, 1111, so 4 bits. So we need a trie where we represent every number in the array as 4 bits.

Next the traversal in the trie takes log(MAX) operations too. For N numbers we do N traversals, so, totally for insertions and traversals, thats 2*N*log(MAX) which is, O(N log(MAX)).

Neat! We have drastically reduced it from O(N²) to O(N log(MAX)).

Algorithm

A[N] = [a1, a2, a3 ..... an]
insert_in_trie(a1)
for i = 2 to N
max_xor_so_far_for_ai = traverse_in_trie_to_get_max_xor(ai)
if max_xor_so_far_for_ai xor ai > max_so_far
max_so_far = max_xor_so_far_for_ai xor ai
insert_in_trie(ai)
print max_so_far

We will explore another similar use case in another problem.