[EMBOSS] accessing emboss ftp site

Hamish McWilliam hpm at bioinfo-user.org.uk
Wed Mar 2 09:25:05 UTC 2011 

Hi Chris,

You had any luck figuring out what is causing this issue with the FTP site?

Hamish

On 29 July 2010 14:46, Hamish McWilliam <hpm at bioinfo-user.org.uk> wrote:
> Hi Chris,
>
>> I did have to turn on support in VSFTPd for "ls -R" because it was
>> considered a possible resource drain or DOS target on large directories but
>> you are right that the extended mode passive stuff does not work. It does
>> not seem to be a firewall issue on the server and I don't think our upstream
>> Juniper SSG intrusion device is interfering.
>
> FWIW I get the same behavior when using passive or "active" FTP modes.
>
>> Can I ask what you are  trying to accomplish? A mirror process? Can we get
>> around this problem via an anonymous RSYNC service pointing at the anonymous
>> FTP root? Passwordless SSH access for a mirror script that uses "rsync -avz
>> -e ssh ..." or similar?
>
> I'm trying to update the EMBL-EBI mirror of the EMBOSS software
> (ftp://ftp.ebi.ac.uk/pub/software/unix/EMBOSS/), this uses Mirror
> (ftp://ftp.ebi.ac.uk/pub/software/unix/mirror.tar.Z) to do the
> updates. As you can tell from the files in the directory this mirror
> no longer works, since it cannot get the listing of files. In fact it
> fails silently so I can't tell when the problem started.
>
>> vsftpd fits our needs for secure anonymous FTP server, if there is an easier
>> route to getting your needs sorted I might want to look at that rather than
>> going down deep into vsftp internals or swapping out the server software.
>
> From a few experiments with our vsftpd setup (CentOS 5.x, vsftpd
> 2.0.5) this looks to be supported by default, other than the -R thing.
>
> From a little Google it could be some interaction with the other
> components (http://www.ncftp.com/ncftpd/doc/misc/ftp_and_firewalls.html)
> where they don't recognize the FTP command correctly and choose the
> wrong mode, but it seems unlikely. Does it work if you run directly
> against the box?
>
> Hamish
>
>> Hamish McWilliam wrote:
>>>
>>> Hi Chris,
>>>
>>>> I can't seem to replicate this at all with any of my available FTP
>>>> clients
>>>> or browser-based FTP clients. It works for me from both Mac OS X clients
>>>> as
>>>> well as a CentOS 5.4 based Linux system.
>>>>
>>>> Is there an FTP client / OS combo that seems in particular not to work?
>>>
>>> This affects all clients I've tried.
>>>
>>>  From your transcript, you are not running the commands required to see
>>> the issue:
>>>
>>> - 'ls' or 'dir' without options works
>>> - 'ls' or 'dir' with options hangs and eventually gives a timeout. FTP
>>> mirror systems often use -lRat as options to 'ls' to get all the files
>>> for a particular branch.
>>>
>>> $ ftp emboss.open-bio.org
>>> Connected to emboss.open-bio.org.
>>> 220 (vsFTPd 2.0.1)
>>> 530 Please login with USER and PASS.
>>> 530 Please login with USER and PASS.
>>> KERBEROS_V4 rejected as an authentication type
>>> Name (emboss.open-bio.org:hpm): ftp
>>> 331 Please specify the password.
>>> Password:
>>> 230 Login successful.
>>> Remote system type is UNIX.
>>> Using binary mode to transfer files.
>>> ftp>  cd /pub/EMBOSS/fixes
>>> 250 Directory successfully changed.
>>> ftp>  ls
>>> 227 Entering Passive Mode (208,94,50,58,51,71)
>>> 150 Here comes the directory listing.
>>> -rw-rw-r--    1 501      503           385 Jul 21 09:25 README.fixes
>>> -rwxrwxr-x    1 501      503        506140 Jul 21 09:25 configure
>>> -rw-rw-r--    1 501      503          5405 Jul 21 09:25 mysql.m4
>>> drwxrwsr-x    4 501      503          4096 Jul 21 09:26 patches
>>> 226 Directory send OK.
>>> ftp>  ls -lRat
>>> 227 Entering Passive Mode (208,94,50,58,31,88)
>>> *HANG*
>>>
>>> Compare with the equivalent session against the EMBL-EBI FTP site:
>>>
>>> $ ftp ftp.ebi.ac.uk
>>> Connected to ftp.ebi.ac.uk.
>>> 220-
>>> 220- ftp1.ebi.ac.uk FTP server
>>> 220-
>>> 220- WARNING: please note that the private part of this ftp service
>>> 220- has been migrated to ftp-private.ebi.ac.uk on 3rd June 2010.
>>> 220
>>> 530 Please login with USER and PASS.
>>> 530 Please login with USER and PASS.
>>> KERBEROS_V4 rejected as an authentication type
>>> Name (ftp.ebi.ac.uk:hpm): ftp
>>> 331 Please specify the password.
>>> Password:
>>> 230 Login successful.
>>> Remote system type is UNIX.
>>> Using binary mode to transfer files.
>>> ftp>  cd /pub/software/unix/EMBOSS/fixes
>>> 250 Directory successfully changed.
>>> ftp>  ls
>>> 227 Entering Passive Mode (193,62,193,4,238,107)
>>> 150 Here comes the directory listing.
>>> -r--r--r--    1 ftp      ftp           180 Jan 15  2010 README.fixes
>>> 226 Directory send OK.
>>> ftp>  ls -lRat
>>> 227 Entering Passive Mode (193,62,193,4,36,65)
>>> 150 Here comes the directory listing.
>>> .:
>>> drwxr-xr-x    3 ftp      ftp          4096 Jan 18  2010 ..
>>> drwxr-xr-x    2 ftp      ftp          4096 Jan 18  2010 .
>>> -r--r--r--    1 ftp      ftp           180 Jan 15  2010 README.fixes
>>> 226 Directory send OK.
>>> ftp>  bye
>>> 221 Goodbye.
>>>
>>> All the best,
>>>
>>> Hamish
>>>
>>>> Here is one example:
>>>>
>>>>> dag-static:~ dag$ ftp emboss.open-bio.org
>>>>> Connected to emboss.open-bio.org.
>>>>> 220 (vsFTPd 2.0.1)
>>>>> Name (emboss.open-bio.org:dag): anonymous
>>>>> 331 Please specify the password.
>>>>> Password:
>>>>> 230 Login successful.
>>>>> Remote system type is UNIX.
>>>>> Using binary mode to transfer files.
>>>>> ftp>  dir
>>>>> 229 Entering Extended Passive Mode (|||54472|)
>>>>> 150 Here comes the directory listing.
>>>>> drwxr-xr-x    8 14       50           4096 May 22  2006 pub
>>>>> 226 Directory send OK.
>>>>> ftp>  cd pub/EMBOSS
>>>>> 250 Directory successfully changed.
>>>>> ftp>  ls
>>>>> 229 Entering Extended Passive Mode (|||38343|)
>>>>> 150 Here comes the directory listing.
>>>>> -rw-rw-r--    1 501      503        389856 Jul 21 09:25
>>>>> CBSTOOLS-1.0.0.tar.gz
>>>>> -rw-rw-r--    1 501      503        426218 Jul 21 09:25
>>>>> DOMAINATRIX-0.1.0.tar.gz
>>>>> -rw-rw-r--    1 501      503        441025 Jul 21 09:25
>>>>> DOMALIGN-0.1.0.tar.gz
>>>>> -rw-rw-r--    1 501      503        452787 Jul 21 09:25
>>>>> DOMSEARCH-0.1.0.tar.gz
>>>>> -rw-rw-r--    1 501      503      23572243 Jul 19 13:41
>>>>> EMBOSS-6.3.1.tar.gz
>>>>> lrwxrwxrwx    1 501      503            19 Jul 19 13:42
>>>>> EMBOSS-latest.tar.gz ->  EMBOSS-6.3.1.tar.gz
>>>>> -rw-rw-r--    1 501      503        373798 Jul 21 09:25 EMNU-1.05.tar.gz
>>>>> -rw-rw-r--    1 501      503        415096 Jul 21 09:25
>>>>> ESIM4-1.0.0.tar.gz
>>>>> -rw-rw-r--    1 501      503        569581 Jul 21 09:25
>>>>> HMMER-2.3.2.tar.gz
>>>>> -rw-rw-r--    1 501      503        350791 Jul 21 09:25
>>>>> IPRSCAN-4.3.1.tar.gz
>>>>> drwxrwsr-x    7 501      503          4096 Feb 01  2006 Jemboss
>>>>> -rw-rw-r--    1 501      503        513418 Jul 21 09:25
>>>>> MEMENEW-4.0.0.tar.gz
>>>>> -rw-rw-r--    1 501      503        823636 Jul 21 09:25
>>>>> MIRA-2.8.2.tar.gz
>>>>> -rw-rw-r--    1 501      503        435315 Jul 21 09:25 MSE-3.0.0.tar.gz
>>>>> -rw-rw-r--    1 501      503        328540 Jul 21 09:25
>>>>> MYEMBOSS-6.3.0.tar.gz
>>>>> -rw-rw-r--    1 501      503        359488 Jul 21 09:25
>>>>> MYEMBOSSDEMO-6.3.0.tar.gz
>>>>> -rw-rw-r--    1 501      503       1667760 Jul 21 09:25
>>>>> PHYLIPNEW-3.69.tar.gz
>>>>> -rw-rw-r--    1 501      503        571008 Jul 21 09:25
>>>>> SIGNATURE-0.1.0.tar.gz
>>>>> -rw-rw-r--    1 501      503        531604 Jul 21 09:25
>>>>> STRUCTURE-0.1.0.tar.gz
>>>>> -rw-rw-r--    1 501      503        386287 Jul 21 09:25
>>>>> TOPO-2.0.0.tar.gz
>>>>> -rw-rw-r--    1 501      503        652433 Jul 21 09:25
>>>>> VIENNA-1.7.2.tar.gz
>>>>> drwxrwsr-x    3 522      503          4096 Aug 21  2006 contrib
>>>>> drwxrwsr-x    2 501      503          4096 Nov 11  2005 doc
>>>>> drwxrwsr-x    3 501      503          4096 Jul 21 09:25 fixes
>>>>> drwxrwsr-x   14 501      503          4096 Jul 19 13:39 old
>>>>> drwxrwsr-x    2 501      503          4096 Jul 06  2005 tutorials
>>>>> drwxrwsr-x    4 501      503          4096 Jul 19 13:36 windows
>>>>> 226 Directory send OK.
>>>>> ftp>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hamish McWilliam wrote:
>>>>>
>>>>> Hi Chris,
>>>>>
>>>>> I'm also seeing problems with the FTP site, but using Mirror rather
>>>>> than Transmit, it looks like the server does not like options being
>>>>> specified to the LIST command:
>>>>>
>>>>> Scanning remote directory /pub/EMBOSS
>>>>> --->    CWD /pub/EMBOSS
>>>>> 250 Directory successfully changed.
>>>>> --->    TYPE A
>>>>> 200 Switching to ASCII mode.
>>>>> --->    PORT 172,21,22,1,171,245
>>>>> 200 PORT command successful. Consider using PASV.
>>>>> --->    PASV
>>>>> 227 Entering Passive Mode (208,94,50,58,104,178)
>>>>> --->    LIST -lat
>>>>> timed out
>>>>> Cannot get remote directory listing because: timed out
>>>>> Cannot get remote directory details (/pub/EMBOSS)
>>>>> disconnecting from emboss.open-bio.org
>>>>>
>>>>> Trying it with a command line client I get the response to hang if I
>>>>> try using any options to ls or dir, without options they are fine.
>>>>>
>>>>> All the best,
>>>>>
>>>>> Hamish
>>>>>
>>>>> On 29 May 2010 03:44, Koen van der Drift<koenvanderdrift at gmail.com>
>>>>>  wrote:
>>>>>>
>>>>>> Hi Chris,
>>>>>>
>>>>>> Did you have a chance to look at this? Just tried again, and Transmit
>>>>>> still
>>>>>> won't let me access the emboss ftp site.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> - Koen.
>>>>>>
>>>>>>
>>>>>> On Apr 23, 2010, at 11:44 AM, Chris Dagdigian wrote:
>>>>>>
>>>>>>> In the last few months the open-bio.org servers switched datacenters,
>>>>>>> IP
>>>>>>> addresses and firewall/IDS appliances. Lots of juicy things to look at
>>>>>>> and
>>>>>>> debug.
>>>>>>>
>>>>>>> Koen - if you have a chance can you send me the IP address that you
>>>>>>> are
>>>>>>> using to connect from? I might be able to find some relevant log
>>>>>>> entries
>>>>>>> with that info.
>>>>>>>
>>>>>>> -Chris
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Koen van der Drift wrote:
>>>>>>>>
>>>>>>>> Just for the record, it used to work with Transmit, this is only from
>>>>>>>> the last few months.
>>>>>>>>
>>>>>>>> - Koen.
>>>>>>>>
>>>>>>>> On Thu, Apr 22, 2010 at 7:28 PM, Chris Dagdigian<dag at sonsorol.org>
>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>> Might be an issue with the Juniper Netscreen firewall/IDS security
>>>>>>>>> appliance
>>>>>>>>> that sits upstream of the EMBOSS FTP server. I'll take a look at the
>>>>>>>>> security logs and alerts.
>>>>>>>>>
>>>>>>>>> -Chris
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Koen van der Drift wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> For a while now I am unable to access the emboss ftp site using the
>>>>>>>>>> OS
>>>>>>>>>> X
>>>>>>>>>> client Transmit. Loggin in works fine, but it chokes on the LIST
>>>>>>>>>> command. I have no problems accessing it from the command line. I
>>>>>>>>>> have
>>>>>>>>>> added the output from Transmit below. I don't know if this is a
>>>>>>>>>> Transmit
>>>>>>>>>> or emboss issue, but just wanted to let you know.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> - Koen.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Transmit 3.6.9 Session Transcript
>>>>>>>>>> LibNcFTP 3.2.1 (August 13, 2007) compiled for UNIX
>>>>>>>>>> Uname: Darwin|exile.local|9.8.0|Darwin Kernel Version 9.8.0: Wed
>>>>>>>>>> Jul
>>>>>>>>>> 15
>>>>>>>>>> 16:57:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_PPC|Power Macintosh
>>>>>>>>>> 220: (vsFTPd 2.0.1)
>>>>>>>>>> Connected to emboss.open-bio.org.
>>>>>>>>>> Cmd: USER anonymous
>>>>>>>>>> 331: Please specify the password.
>>>>>>>>>> Cmd: PASS NcFTP@
>>>>>>>>>> 230: Login successful.
>>>>>>>>>> Cmd: TYPE A
>>>>>>>>>> 200: Switching to ASCII mode.
>>>>>>>>>> Logged in to emboss.open-bio.org as anonymous.
>>>>>>>>>> Cmd: SYST
>>>>>>>>>> 215: UNIX Type: L8
>>>>>>>>>> Cmd: PWD
>>>>>>>>>> 257: "/"
>>>>>>>>>> Cmd: CWD /pub/EMBOSS/fixes
>>>>>>>>>> 250: Directory successfully changed.
>>>>>>>>>> Cmd: PWD
>>>>>>>>>> 257: "/pub/EMBOSS/fixes"
>>>>>>>>>> Cmd: PASV
>>>>>>>>>> 227: Entering Passive Mode (208,94,50,58,83,232)
>>>>>>>>>> Cmd: LIST -a
>>>>>>>>>> Could not read reply from control connection -- timed out.
>>>>>>>>>> (SReadline
>>>>>>>>>> 1)
>>>>>>>>>> 220: (vsFTPd 2.0.1)
>>>>>>>>>> Connected to emboss.open-bio.org.
>>>>>>>>>> Cmd: USER anonymous
>>>>>>>>>> 331: Please specify the password.
>>>>>>>>>> Cmd: PASS NcFTP@
>>>>>>>>>> 230: Login successful.
>>>>>>>>>> Logged in to emboss.open-bio.org as anonymous.
>>>>>>>>>> Cmd: SYST
>>>>>>>>>> 215: UNIX Type: L8
>>>>>>>>>> Cmd: PWD
>>>>>>>>>> 257: "/"
>>>>>>>>>> Cmd: CWD /pub/EMBOSS/fixes
>>>>>>>>>> 250: Directory successfully changed.
>>>>>>>>>> Cmd: PWD
>>>>>>>>>> 257: "/pub/EMBOSS/fixes"
>>>>>>>>>> Cmd: PASV
>>>>>>>>>> 227: Entering Passive Mode (208,94,50,58,222,100)
>>>>>>>>>> Cmd: LIST -a
>>>>>>>>>> Could not read reply from control connection -- timed out.
>>>>>>>>>> (SReadline
>>>>>>>>>> 1)
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> EMBOSS mailing list
>>>>>>>>>> EMBOSS at lists.open-bio.org
>>>>>>>>>> http://lists.open-bio.org/mailman/listinfo/emboss
>>>>>>
>>>>>> _______________________________________________
>>>>>> EMBOSS mailing list
>>>>>> EMBOSS at lists.open-bio.org
>>>>>> http://lists.open-bio.org/mailman/listinfo/emboss
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>>
>
>
>
> --
> ----
> "Saying the internet has changed dramatically over the last five years
> is cliché – the internet is always changing dramatically" - Craig
> Labovitz, Arbor Networks.
>



-- 
----
"Saying the internet has changed dramatically over the last five years
is cliché – the internet is always changing dramatically" - Craig
Labovitz, Arbor Networks.

More information about the EMBOSS mailing list