During his 15-year run in the securities business, Kenneth Dwyer peddled stock for a half dozen firms that were later booted from the business by regulators.

Collectively, the shuttered and defunct firms that Dwyer worked for had left thousands of investors with alleged losses and unpaid claims totaling more than $85 million, according to court documents and lawyers.

Our analysis of never-before compiled industry regulatory records revealed that Dwyer was one of more than 5,000 brokers who were still selling securities in early 2013 after working for firms expelled by regulators since 2005. We found 610 brokers who, like Dwyer, were still in the business after having worked at more than one such firm. On Wall Street, there is a term for brokers shuttling between troubled firms: “cockroaching.”

We found these brokers averaged more than eight times as many arbitration claims and other required regulatory disclosures as their peers. Nearly 25 percent of them had three or more such black marks, compared to just 1.8 percent for the industry overall.

To undertake our investigation into stockbroker-migratory patterns, Wall Street Journal reporter Jean Eaglesham and I faced a critical challenge stemming from laws passed in the wake of the Great Depression: The disciplinary records and work histories of the more than 630,000 people licensed to sell securities in America are not public.

In the fallow years of the 1930s, lawmakers came together to reshape the way in which the country’s equity markets are regulated. The primary task of overseeing the nation’s stockbrokers fell to a new type of entity, one whose membership consisted of the very people it was tasked with overseeing. This new type of body was called a self-regulatory organization and in order to sell stock, brokers would have to join. Today, the main SRO is called the Financial Industry Regulatory Authority.

FINRA maintains a database called the Central Registration Depository, which contains an array of regulatory information, including the work and disciplinary histories of every one of its member brokers. But because FINRA isn’t a government agency, its books and records aren’t subject to freedom of information act requests. While the public can search for an individual broker’s CRD record online, the organization won’t turn over the bulk data contained in its CRD system.

To get around this barrier, Eaglesham and I turned to the states.

Although stockbrokers register with FINRA nationally, they’re also required to register in each of the states where they plan to sell stock. The local state securities regulators are then charged with co-regulating brokers and have bulk access to the slice of the CRD records containing brokers registered with them.

By filing public records requests with all 50 of these regulators, Eaglesham and I were able to stitch together a nearly complete picture of stockbrokers across the country. Because brokers can register in multiple states, we focused our budget and attention on the most populous ones. Thus, we agreed to pay a hefty records fee for data from New York, but when Hawaii asked for the same amount, we turned them down.

In the process we came across several reluctant securities regulators. Some were wary of the technical work involved. To assist them, we found a particularly tech savvy regulator and had him walk us through the process of extracting CRD data in bulk. We then included his procedure in our future FOIAs.

Other regulators were wary about turning over their data. A lawyer at one large state responded to our FOIA by saying “states do not own or keep the data on the CRD,” and insisting that “FINRA owns and keeps these records,” and “This has nothing to do with the Freedom of Information Law, which does not require state agencies to obtain data from federal agencies, such as FINRA.”

This response was deeply flawed for several reasons, not the least of which is the fact that FINRA is not a federal agency. Fortunately for our story, we were able to find a 2007 court filing by this same agency. It argued that “all state securities authorities have an ownership interest in CRD information,” and that “CRD records are not only co-owned by the states, they also constitute ‘original’ records of the states.” We obtained the data we wanted shortly after bringing this to their attention.

The data itself is spread across roughly 20 tables linked together by a primary identifier – called a CRD number – assigned to each broker.

As our records poured in, we were confronted with a new hurdle. While most regulators sent all the tables we’d asked for, the formats they used differed wildly. Some states gave us data in nested XML form. Others used comma-separated files. Still others delivered the data in Excel or Access. Even field names were inconsistent. Some regulators called the CRD number “individualcrdnumber,” but others instead called it “Individual_CRDNb” and “Individual CRD #.”

All in all, we received data from 21 states spread across 410 tables consisting of 13,301 fields. We set about rebuilding as much of the underlying CRD database as we could. We wrote fairly extensive PHP, Python and C# scripts that imported the various file types, standardized field names and values (like dates) and funneled everything into centralized tables.

In the end, we were able to compile records on 558,245 brokers, or about 88 percent of the total number of brokers working in America.

The idea of examining migratory patterns came quickly. Both Eaglesham and I had heard rumors of brokerages stamped out by regulators only to re-open the next day with a new sign on the door.

We were able to highlight connections between brokers and the firms where they worked as they traveled through the business using social network analysis tools such as the open-source Gephi. We found clusters of brokers migrating together, and found some groups had much worse disciplinary records than others.

Our reporting revealed that some of the most troubled brokers had indeed worked at firms that were later shuttered by regulators. So we decided to see how many brokers were still in business after working at closed-down firms. FINRA claimed not to have a list of brokerages booted from the business, so we combed through 105 of the regulator’s monthly reports between 2005 and 2012 and plucked each instance in which it had touted an expulsion.

We then cross-referenced this list with our brokers’ work histories. A huge number of these troubled brokers worked in Long Island. Eaglesham and I traipsed across the peninsula tracking down as many brokers as we could, put together our story and published in October.

The reaction was swift. In January, under pressure from lawmakers, FINRA convened a six-member team tasked with reviewing brokers with extensive disciplinary histories and troubled migratory patterns. FINRA also announced plans to deploy a new tool called the Broker Migration Model that will track the movement of brokers between expelled firms.

More recently, we used the data to find brokers with unreported bankruptcies and criminal charges and show how brokers who repeatedly failed the entrance exam had worse disciplinary records.


Follow Wall Street Journal reporter Rob Barry on Twitter @rob_barry.